Install Bigdesk Plugin for elasticsearch.

Bigdesk

        Live charts and statistics for elasticsearch cluster.

        This plugin works only for elasticsearch 1.0.0 or higher:

• Go to elastic search home directory: 
Generally ES_HOME is /usr/share/elasticsearch
cd  /usr/share/elasticsearch                                                                                                                           
• Navigate to the node installation folder and run the following command: 
$  ./bin/plugin -install lukas-vlcek/bigdesk                                                                                                     
• To install a specific version (for example 2.4.0) run: 
$  ./bin/plugin -install lukas-vlcek/bigdesk/2.4.0                                                                                                
• Then you can navigate your web browser to http://<elasticsearch_REST_endpoint>/_plugin/bigdesk/

for example in case of localhost:
http://localhost:9200/_plugin/bigdesk/#cluster          

Ref: http://bigdesk.org/

     

Nodes shutdown commands in elasticsearch.

nodes shutdown

The nodes shutdown API allows to shutdown one or more (or all) nodes in the cluster. Here is an example of shutting the _local node the request is directed to:

$ curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'

Specific node(s) can be shutdown as well using their respective node ids (or other selective options as explained here .):

$ curl -XPOST 'http://localhost:9200/_cluster/nodes/nodeId1,nodeId2/_shutdown'

The master (of the cluster) can also be shutdown using:

$ curl -XPOST 'http://localhost:9200/_cluster/nodes/_master/_shutdown'

Finally, all nodes can be shutdown using one of the options below:

$ curl -XPOST 'http://localhost:9200/_shutdown'

$ curl -XPOST 'http://localhost:9200/_cluster/nodes/_shutdown'

$ curl -XPOST 'http://localhost:9200/_cluster/nodes/_all/_shutdown'

delay

By default, the shutdown will be executed after a 1 second delay (1s). The delay can be customized by setting the delay parameter in a time value format. For example:

$ curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown?delay=10s'

disable shutdown

The shutdown API can be disabled by setting action.disable_shutdown in the node configuration.

Ref: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-shutdown.html

How to install beaver logstash client?

Install dependencies for beaver:

sudo apt-get install python-dev

sudo apt-get install gcc

sudo apt-get install python-pip

Install Beaver:

sudo pip install beaver==18                   or

sudo pip install beaver==28

Run Beaver:

sudo beaver -c ./beaver1.conf -t redis

beaver.conf

[beaver]

transport: redis

redis_url: redis://127.0.0.1:6379/0

redis_namespace: logstash_test

ssh_key_file: /home/----------/.ssh/id_rsa

ssh_tunnel: ubuntu@xyz

ssh_tunnel_port: 6379

ssh_remote_host: 127.0.0.1

ssh_remote_port: 6379

[/var/log/*.log]

type: production

Start sending logs from beaver to redis:

beaver -c ./beaver.conf -t redis   

How to add remove user (standard user/non-root) in Kali Linux and add them to sudoer ?

Add user in Kali Linux or other Linux:

• First of all let’s confirm which version of Linux and Kernel I’m running.In command prompt type in

• uname –a
  lsb_release –a

  

• Now let’s add user. Open terminal and type following to create new user (replace user1 with your desired user name)

  useradd -m user1

  
  (Note: -m means create home directory which is usually /home/username)

• Now set password for this user

  passwd user1

  Enter desired password twice
  

• Add user to sudo group (to allow user to install software, allow printing, use privileged mode etc.)

  usermod -a -G sudo user1

  
  (Note: -a means append or add and –G mean to specified group/groups)

• Change default shell of previously created user to bash

  chsh -s /bin/bash user1

  
  (Note: chsh mean change login shell, -s is the name of the specified shell you want for the user, in this case /bin/bash)
  Nice, all worked out as expected.
  Let’s logout and login back as our new Standard Non-root user (user1)

Login as new user

• Once logged in, let’s confirm from command line who I really am! In terminal type in the following

  whoami

  
  Take note of user1@kali prompt. That also confirms who you are.
• And let’s check my group affinity, type in the following in terminal:

  groups

  
  Looks good to me so far.
  I am part of user1 group (my primary) and sudo group. That means I can run privileged commands or just become root myself if required.
• Become root!

  sudo su –

  and type in user1’s password to become root.
  
  See the prompt becomes root@kali instead of user1@kali. That means you’re now root and get to run anything Kali got to offer.
• Lets confirm that using whoami command

  whoami

  
  So far so good. Now how do you delete a user?

Delete user in Kali Linux:

• Log in as root user again. Open terminal and type:

  userdel –r user1

  (Note: -r means delete all files and home directory for user1)
  You can replace user1 with your desired user name.
  
  I have an error “user1 is currently used by process 5866”.
  So process ID 5866 is being used by user1. (I know that it’s the gnome-keyring process running in background when I used sudo su – command earlier. (Gnome-Keyring error is quite common in Debian when you install multiple Windows or Desktop Managers. There’s a separate guide see How to fix Gnome-Keyring error. This demonstrates how to fix gnome-keyring error for root user). In our case we are not worried as we just want to delete that user.
  Let do that.
• Type in the following in terminal to kill the process used by user1.

  kill -9 5866

  This kills the process immediately.
  
  (Note: don’t kill root or system processes unless you know what you’re doing)
• Now let’s try to delete the user again.

  userdel –r user1

  
  We have a message. “userdel: user1 mail spool (/var/mail/user1) not found”.
  (Note: -r means delete all files and home directory for user1)
  Are we going to worry about it? Not really, we never created a mailbox for user1.
• Just to confirm everything for user1 was deleted list files in home directory

  ls /home

  
  Nothing.. that’s good news, all files and folders were deleted for user1.
• Want to double-check?

  su user1

  
  Perfect user1 was deleted successfully.

Ref: http://www.blackmoreops.com/2013/11/27/how-to-add-remove-user-standard-user-non-root-in-kali-linux/

cluster health in elasticsearch

cluster health

The cluster health API allows to get a very simple status on the health of the cluster.

$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "testcluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

The API can also be executed against one or more indices to get just the specified indices health:

$ curl -XGET 'http://localhost:9200/_cluster/health/test1,test2'

The cluster health status is: green, yellow or red. On the shard level, a red status indicates that the specific shard is not allocated in the cluster, yellow means that the primary shard is allocated but replicas are not, and green means that all shards are allocated. The index level status is controlled by the worst shard status. The cluster status is controlled by the worst index status.

One of the main benefits of the API is the ability to wait until the cluster reaches a certain high water-mark health level. For example, the following will wait for 50 seconds for the cluster to reach the yellow level (if it reaches the green or yellow status before 50 seconds elapse, it will return at that point):

$ curl -XGET 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s'

request parameters

The cluster health API accepts the following request parameters:

level

Can be one of cluster, indices or shards. Controls the details level of the health information returned. Defaults to cluster.

wait_for_status

One of green, yellow or red. Will wait (until the timeout provided) until the status of the cluster changes to the one provided. By default, will not wait for any status.

wait_for_relocating_shards

A number controlling to how many relocating shards to wait for. Usually will be 0 to indicate to wait till all relocation have happened. Defaults to not to wait.

wait_for_nodes

The request waits until the specified number N of nodes is available. It also accepts >=N, <=N, >N and <N. Alternatively, it is possible to use ge(N), le(N), gt(N) and lt(N)notation.

timeout

A time based parameter controlling how long to wait if one of the wait_for_XXX are provided. Defaults to 30s.

The following is an example of getting the cluster health at the shards level:

$ curl -XGET 'http://localhost:9200/_cluster/health/twitter?level=shards'

Ref: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html

Nodes info in elasticsearch

nodes info

The cluster nodes info API allows to retrieve one or more (or all) of the cluster nodes information.

curl -XGET 'http://localhost:9200/_nodes'
curl -XGET 'http://localhost:9200/_nodes/nodeId1,nodeId2'

The first command retrieves information of all the nodes in the cluster. The second command selectively retrieves nodes information of only nodeId1 and nodeId2. All the nodes selective options are explained here.

By default, it just returns all attributes and core settings for a node. It also allows to get only information on settings, os, process, jvm, thread_pool, network, transport, httpand plugins:

curl -XGET 'http://localhost:9200/_nodes/process'
curl -XGET 'http://localhost:9200/_nodes/_all/process'
curl -XGET 'http://localhost:9200/_nodes/nodeId1,nodeId2/jvm,process'
# same as above
curl -XGET 'http://localhost:9200/_nodes/nodeId1,nodeId2/info/jvm,process'

curl -XGET 'http://localhost:9200/_nodes/nodeId1,nodeId2/_all

The _all flag can be set to return all the information - or you can simply omit it.

plugins - if set, the result will contain details about the loaded plugins per node:

• name: plugin name
• description: plugin description if any
• site: true if the plugin is a site plugin
• jvm: true if the plugin is a plugin running in the JVM
• url: URL if the plugin is a site plugin

The result will look similar to:

{
  "cluster_name" : "test-cluster-MacBook-Air-de-David.local",
  "nodes" : {
    "hJLXmY_NTrCytiIMbX4_1g" : {
      "name" : "node4",
      "transport_address" : "inet[/172.18.58.139:9303]",
      "hostname" : "MacBook-Air-de-David.local",
      "version" : "0.90.0.Beta2-SNAPSHOT",
      "http_address" : "inet[/172.18.58.139:9203]",
      "plugins" : [ {
        "name" : "test-plugin",
        "description" : "test-plugin description",
        "site" : true,
        "jvm" : false
      }, {
        "name" : "test-no-version-plugin",
        "description" : "test-no-version-plugin description",
        "site" : true,
        "jvm" : false
      }, {
        "name" : "dummy",
        "description" : "No description found for dummy.",
        "url" : "/_plugin/dummy/",
        "site" : false,
        "jvm" : true
      } ]
    }
  }
}

if your plugin data is subject to change use plugins.info_refresh_interval to change or disable the caching interval:

# Change cache to 20 seconds
plugins.info_refresh_interval: 20s
# Infinite cache
plugins.info_refresh_interval: -1
# Disable cache
plugins.info_refresh_interval: 0

Ref: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-info.html

Nodes stats in elasticsearch

nodes stats

nodes statistics

The cluster nodes stats API allows to retrieve one or more (or all) of the cluster nodes statistics.

curl -XGET 'http://localhost:9200/_nodes/stats'
curl -XGET 'http://localhost:9200/_nodes/nodeId1,nodeId2/stats'

The first command retrieves stats of all the nodes in the cluster. The second command selectively retrieves nodes stats of only nodeId1 and nodeId2. All the nodes selective options are explainedhere.

By default, all stats are returned. You can limit this by combining any of indices, os, process, jvm, network, transport, http, fs, breaker and thread_pool. For example:

indices	Indices stats about size, document count, indexing and deletion times, search times, field cache size , merges and flushes
fs	File system information, data path, free disk space, read/write stats
http	HTTP connection information
jvm	JVM stats, memory pool information, garbage collection, buffer pools
network	TCP information
os	Operating system stats, load average, cpu, mem, swap
process	Process statistics, memory consumption, cpu usage, open file descriptors
thread_pool	Statistics about each thread pool, including current size, queue and rejected tasks
transport	Transport statistics about sent and received bytes in cluster communication
breaker	Statistics about the field data circuit breaker

# return indices and os
curl -XGET 'http://localhost:9200/_nodes/stats/os'
# return just os and process
curl -XGET 'http://localhost:9200/_nodes/stats/os,process'
# specific type endpoint
curl -XGET 'http://localhost:9200/_nodes/stats/process'
curl -XGET 'http://localhost:9200/_nodes/10.0.0.1/stats/process'

The all flag can be set to return all the stats.

field data statistics

You can get information about field data memory usage on node level or on index level.

# Node Stats
curl localhost:9200/_nodes/stats/indices/field1,field2?pretty
# Indices Stat
curl localhost:9200/_stats/fielddata/field1,field2?pretty
# You can use wildcards for field names
curl localhost:9200/_stats/fielddata/field*?pretty
curl localhost:9200/_nodes/stats/indices/field*?pretty

Ref: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html