Friday, July 7, 2017

Log Analysis and Alert system with ELK and Elastalert



This document will show the setup and configuration required for running the logstash, elasticsearch, kibana, and elastalert for alerting.

This document shows the setup on MacOS, assuming you have homebrew package manager.
 Step1: Run "brew install logstash"
 Step2: Run  "brew install elasticsearch"
 Step3: Run "brew install kibana"

This example will consider the selenium server logs and display them on Kibana and alert them by querying the elasticsearch using elastalert.

Run Selenium Server:
1. cd Downloads
2. nohup java -jar -Dselenium.LOGGER=output.log -Dselenium.LOGGER.level=FINEST selenium-server-standalone-3.4.0.jar &

Now, we will see an output.log file generated in the Downloads directory.

Configure Logstash:
  1. Create a logstash configuration file in "Downloads" directory as follows
input {

  file {
     path => "/Users/Yuvaraj/Downloads/output.log"
     start_position => beginning
  }
}
filter {
  grok {
     match => { "message" => "%{COMBINEDAPACHELOG}"}
  }
  geoip {
     source => "clientip"
  }
}
output {
  elasticsearch {

  }
}

and save it as selenium-log.conf.
2.Now run the logstash with this configuration as below in the "Downloads" directory.
    logstash -f selenium-log.conf

Now logstash should display the following logs
Sending Logstash's logs to /usr/local/Cellar/logstash/5.4.2/libexec/logs which is now configured via log4j2.properties
[2017-07-06T09:34:36,967][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2017-07-06T09:34:36,972][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-06T09:34:37,050][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xeda20da URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused"}
[2017-07-06T09:34:37,051][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-07-06T09:34:37,056][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused {:url=>http://127.0.0.1:9200/, :error_message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2017-07-06T09:34:37,057][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :backtrace=>["/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:271:in `perform_request_to_url'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:257:in `perform_request'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:347:in `with_connection'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:256:in `perform_request'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:264:in `get'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `get_version'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:16:in `get_es_version'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:20:in `get_es_major_version'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in `install_template'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/common.rb:62:in `install_template'", "/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `register'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/output_delegator.rb:41:in `register'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugin'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/pipeline.rb:279:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/pipeline.rb:279:in `register_plugins'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/pipeline.rb:288:in `start_workers'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/pipeline.rb:214:in `run'", "/usr/local/Cellar/logstash/5.4.2/libexec/logstash-core/lib/logstash/agent.rb:398:in `start_pipeline'"]}
[2017-07-06T09:34:37,059][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x289e630a URL://127.0.0.1>]}
[2017-07-06T09:34:37,109][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/local/Cellar/logstash/5.4.2/libexec/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.1.1-java/vendor/GeoLite2-City.mmdb"}
[2017-07-06T09:34:37,132][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-07-06T09:34:37,254][INFO ][logstash.pipeline        ] Pipeline main started
[2017-07-06T09:34:37,295][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-07-06T09:34:42,071][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-06T09:34:42,076][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xeda20da URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused"}
[2017-07-06T09:34:47,092][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-06T09:34:47,097][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xeda20da URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused"}
[2017-07-06T09:34:52,112][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-06T09:34:52,116][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xeda20da URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused"}
[2017-07-06T09:34:57,127][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-06T09:34:57,129][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xeda20da URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused"}
[2017-07-06T09:35:02,138][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-07-06T09:35:02,140][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xeda20da URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused"}
[2017-07-06T09:35:07,162][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}

[2017-07-06T09:35:07,269][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0xeda20da URL:http://127.0.0.1:9200/>}

Now, you can confirm if the logstash is running by

curl -i http://localhost:9600/

Response: 200
{"host":"Z4437-E6B0-1CA8.uk.ad.ba.com","version":"5.4.2","http_address":"127.0.0.1:9600","id":"2b161c1d-ac74-4840-b716-67577b2c03d7","name":"Z4437-E6B0-1CA8.uk.ad.ba.com","build_date":"2017-06-15T03:03:41Z","build_sha":"76c4926f4da6fa4a5d1ec9f5e5770dd85a8b1958","build_snapshot":false}

Configure & Run Elastic search:
1. Run the elastic search with the following command
    cd Downloads
    elasticsearch
Now, we must see the following logs
[2017-07-06T09:34:56,207][INFO ][o.e.n.Node               ] [] initializing ...
[2017-07-06T09:34:56,287][INFO ][o.e.e.NodeEnvironment    ] [4_D7gOx] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [59.7gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2017-07-06T09:34:56,288][INFO ][o.e.e.NodeEnvironment    ] [4_D7gOx] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-06T09:34:56,351][INFO ][o.e.n.Node               ] node name [4_D7gOx] derived from node ID [4_D7gOxPQZCwvn8sll6G4A]; set [node.name] to override
[2017-07-06T09:34:56,352][INFO ][o.e.n.Node               ] version[5.4.3], pid[2569], build[eed30a8/2017-06-22T00:34:03.743Z], OS[Mac OS X/10.11.6/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_51/25.51-b03]
[2017-07-06T09:34:56,352][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/local/Cellar/elasticsearch/5.4.3/libexec]
[2017-07-06T09:34:57,121][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [aggs-matrix-stats]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [ingest-common]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [lang-expression]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [lang-groovy]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [lang-mustache]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [lang-painless]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [percolator]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [reindex]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [transport-netty3]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] loaded module [transport-netty4]
[2017-07-06T09:34:57,122][INFO ][o.e.p.PluginsService     ] [4_D7gOx] no plugins loaded
[2017-07-06T09:34:58,526][INFO ][o.e.d.DiscoveryModule    ] [4_D7gOx] using discovery type [zen]
[2017-07-06T09:34:59,105][INFO ][o.e.n.Node               ] initialized
[2017-07-06T09:34:59,106][INFO ][o.e.n.Node               ] [4_D7gOx] starting ...
[2017-07-06T09:34:59,254][INFO ][o.e.t.TransportService   ] [4_D7gOx] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2017-07-06T09:35:02,301][INFO ][o.e.c.s.ClusterService   ] [4_D7gOx] new_master {4_D7gOx}{4_D7gOxPQZCwvn8sll6G4A}{gGc_p74URDm6iT5_rczMOg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-07-06T09:35:02,319][INFO ][o.e.h.n.Netty4HttpServerTransport] [4_D7gOx] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2017-07-06T09:35:02,321][INFO ][o.e.n.Node               ] [4_D7gOx] started
[2017-07-06T09:35:02,688][INFO ][o.e.g.GatewayService     ] [4_D7gOx] recovered [14] indices into cluster_state
[2017-07-06T09:35:03,391][INFO ][o.e.c.r.a.AllocationService] [4_D7gOx] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2017.06.30][4]] ...]).
[2017-07-06T09:36:22,753][INFO ][o.e.c.m.MetaDataCreateIndexService] [4_D7gOx] [logstash-2017.07.06] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]
[2017-07-06T09:36:22,934][INFO ][o.e.c.m.MetaDataMappingService] [4_D7gOx] [logstash-2017.07.06/9Y3_2-ImRP6F3P0DQH61Bg] create_mapping [logs]
[2017-07-06T12:07:48,059][INFO ][o.e.c.m.MetaDataMappingService] [4_D7gOx] [elastalert_status/x3wnpwI-QnSWrDFbBrPh4A] update_mapping [silence]
[2017-07-06T12:17:25,204][INFO ][o.e.c.m.MetaDataMappingService] [4_D7gOx] [elastalert_status/x3wnpwI-QnSWrDFbBrPh4A] update_mapping [elastalert_error]
[2017-07-06T12:17:25,219][INFO ][o.e.c.m.MetaDataMappingService] [4_D7gOx] [elastalert_status/x3wnpwI-QnSWrDFbBrPh4A] update_mapping [elastalert]
[2017-07-06T13:49:19,911][INFO ][o.e.c.m.MetaDataMappingService] [4_D7gOx] [elastalert_status/x3wnpwI-QnSWrDFbBrPh4A] update_mapping [elastalert]
[2017-07-07T09:36:51,669][INFO ][o.e.c.m.MetaDataCreateIndexService] [4_D7gOx] [logstash-2017.07.07] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]

[2017-07-07T09:36:51,744][INFO ][o.e.c.m.MetaDataMappingService] [4_D7gOx] [logstash-2017.07.07/GldhiF5YRbCLVY3FoVidpw] create_mapping [logs]

We can confirm if the elasticsearch is running by making the following request
Z4437-E6B0-1CA8:Downloads Yuvaraj$ curl -i http://localhost:9200/
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 335

{
  "name" : "4_D7gOx",
  "cluster_name" : "elasticsearch_Yuvaraj",
  "cluster_uuid" : "5aVFyKihT9O61L1XdEopbA",
  "version" : {
    "number" : "5.4.3",
    "build_hash" : "eed30a8",
    "build_date" : "2017-06-22T00:34:03.743Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.1"
  },
  "tagline" : "You Know, for Search"

}

Configure & Run Kibana:
1. cd Downloads
    kibana
  log   [12:58:27.113] [info][status][plugin:kibana@5.4.2] Status changed from uninitialized to green - Ready
  log   [12:58:27.190] [info][status][plugin:elasticsearch@5.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [12:58:27.223] [info][status][plugin:console@5.4.2] Status changed from uninitialized to green - Ready
  log   [12:58:27.232] [warning] You're running Kibana 5.4.2 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v5.4.3 @ 127.0.0.1:9200 (127.0.0.1)
  log   [12:58:27.243] [info][status][plugin:metrics@5.4.2] Status changed from uninitialized to green - Ready
  log   [12:58:27.403] [info][status][plugin:elasticsearch@5.4.2] Status changed from yellow to green - Kibana index ready
  log   [12:58:27.404] [info][status][plugin:timelion@5.4.2] Status changed from uninitialized to green - Ready
  log   [12:58:27.408] [info][listening] Server running at http://localhost:5601

  log   [12:58:27.409] [info][status][ui settings] Status changed from uninitialized to green - Ready

Now, launch "http://localhost:5601" in your favourite browser. You will the following web app

With this, you have completed running the log analysis tools, once you configure the index pattern, you can see the logs by clicking the discovery

Configure Elastalert to monitor the logs and to alert:
1. Install the elastalert with the python as per the documentation available
https://elastalert.readthedocs.io/en/latest/running_elastalert.html
2. Once the installation is completed, run the elastalert after creating the following configurations
    Elastalert needs a global config.yaml which it refers always. So, create a config.yaml as follows
es_host: localhost
es_port: 9200
smtp_host: smtp.gmail.com
email: gyuvaraj16@gmail.com
smtp_port: 465
smtp_ssl: true
smtp_auth_file: '/Users/Yuvaraj/Downloads/smtp_auth_file.yaml'
from_addr: gyuvaraj10@gmail.com
rules_folder: elastrules 
buffer_time: 
  hours: 1000
run_every: 
  minutes: 1

writeback_index: elastalert_status 

Now, create a rule & alert configuration yaml file as follows
mkdir elastrules
cd elastrules
vim selenium-rule.yaml

alert: 
  - "command"
command: "echo {match[username]}"
email: 
  - "yuvaraj.gunisetti@ba.com"
es_host: localhost
es_port: 9200
filter: 
  - query: 
      query_string:
          query: "message: com.opera.core.systems.OperaDriver"
index: logstash-*
name: Selenium_Alert
num_events: 1
timeframe: 
  hours: 1000
type: frequency
realert:

  minutes: 0

Now, go to Downloads directory where the config.yaml is available and run the following command

python -m elastalert.elastalert --verbose --rule elastrules/selenim-rule.yaml --es_debug_trace file.log 

Now,if there are any matches with the pattern specified in the selenium-rule.yaml "filter" you will see the following logs
INFO:elastalert:Queried rule Selenium_Alert from 2017-07-06 13:52 BST to 2017-07-07 09:37 BST: 60 / 60 hits
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}
{match[username]}

INFO:elastalert:Ran Selenium_Alert from 2017-07-06 13:52 BST to 2017-07-07 09:37 BST: 60 query hits (28 already seen), 32 matches, 32 alerts sent


You are done



Tuesday, June 6, 2017

Link Godaddy DNS with AWS Elastic IP




First we need to set up AWS to provide an IP address for your DNS settings.
  1. On EC2 Management console you will have a vertical menu on the left hand side.
  2. Under “NETWORK & SECURITY” group click on “Elastic IPs”.
  3. On the top menu you will see a blue button “Allocate New Address” click on it.
  4. Just be sure “EIP used in” is set to “EC2” then click “Yes, Allocate”.
  5. A new IP address will be created on the table, select it by clicking on the empty square by the left of the name.
  6. Now click on “Associate Address” on the pop-up click on instance and select the instance you would like to associate to this IP.
  7. Finally click “Associate” and that’s it. For now to access via SSH, FTP, etc. you will need to use the new elastic IP.
On the godaddy side we will set up the points to address with the new elastic ip.
  1. Login into your godaddy account.
  2. Under the upper menu click “Domains” and then click “Manage my Domains”.
  3. Select the domain you would like to change by clicking the link to the domain on the table under “Domain Name” column.
  4. In Domain Details there are three tabs, you should click on “DNS Zone File”.
  5. Under A(Host) , click on “Edit Record” at the end in “Actions” column.
  6. Now change the value on the field “Points to” with the elastic ip of your amazon ec2 instance.

Monday, May 15, 2017

Load balance multiple selenium Grid hubs for Distributed testing




Load balance, your Selenium Grid hub servers to improve the testing speed. There are multiple providers available in the market to load balance your applications. Few open source softwares are
nginx, apache httpd, etc.,

This page will help you in setting up your selenium grid hub servers in a load balanced environment by using the nginx.

1. As a primary step install the nginx on your local machine. Those who use mac can use the home brew to install the nginx.
  command: brew install nginx
if the nginx is already installed and needs to be upgraded, then run "brew update nginx"

2. Run <ps -ef|grep "nginx"> to check if there are any nginx process running in the background.
   If you find any nginx process running, then either just stop the service or kill the process

3. Now, download the selenium-server-standalone jar file from the seleniumhq.org. This example uses "selenium-server-standalone-2.53.1.jar".

4. From the server directory, run the hub as follows


java -jar selenium-server-standalone-2.53.1.jar -role hub   --- call it ---- HUB A

17:24:16.808 INFO - Launching Selenium Grid hub
2017-05-15 17:24:17.359:INFO::main: Logging initialized @691ms
17:24:17.367 INFO - Will listen on 4444
17:24:17.402 INFO - Will listen on 4444
2017-05-15 17:24:17.405:INFO:osjs.Server:main: jetty-9.2.z-SNAPSHOT
2017-05-15 17:24:17.425:INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler@a74868d{/,null,AVAILABLE}
2017-05-15 17:24:17.445:INFO:osjs.ServerConnector:main: Started ServerConnector@5c072e3f{HTTP/1.1}{0.0.0.0:4444}
2017-05-15 17:24:17.446:INFO:osjs.Server:main: Started @779ms
17:24:17.446 INFO - Nodes should register to http://169.254.86.238:4444/grid/register/
17:24:17.446 INFO - Selenium Grid hub is up and running
17:40:18.394 INFO - Registered a node http://169.254.86.238:5555

5. From the same directory, create another hub as follows

java -jar selenium-server-standalone-2.53.1.jar -role hub -port 4445 ---- call it ---- HUB B

17:25:00.741 INFO - Launching Selenium Grid hub
2017-05-15 17:25:01.296:INFO::main: Logging initialized @696ms
17:25:01.304 INFO - Will listen on 4445
17:25:01.344 INFO - Will listen on 4445
2017-05-15 17:25:01.346:INFO:osjs.Server:main: jetty-9.2.z-SNAPSHOT
2017-05-15 17:25:01.371:INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler@a74868d{/,null,AVAILABLE}
2017-05-15 17:25:01.401:INFO:osjs.ServerConnector:main: Started ServerConnector@5c072e3f{HTTP/1.1}{0.0.0.0:4445}
2017-05-15 17:25:01.401:INFO:osjs.Server:main: Started @801ms
17:25:01.402 INFO - Nodes should register to http://169.254.86.238:4445/grid/register/
17:25:01.402 INFO - Selenium Grid hub is up and running
17:42:49.934 INFO - Registered a node http://169.254.86.238:5556
 
6. Now register a node to hub A as follows,


java -jar selenium-server-standalone-2.53.1.jar -role wd -hub http://localhost:4444/grid/register

17:40:17.905 INFO - Launching a Selenium Grid node
17:40:18.258 INFO - Java: Oracle Corporation 25.51-b03
17:40:18.258 INFO - OS: Mac OS X 10.11.6 x86_64
17:40:18.265 INFO - v2.53.1, with Core v2.53.1. Built from revision a36b8b1
17:40:18.316 INFO - Driver provider org.openqa.selenium.ie.InternetExplorerDriver registration is skipped:
registration capabilities Capabilities [{ensureCleanSession=true, browserName=internet explorer, version=, platform=WINDOWS}] does not match the current platform MAC
17:40:18.316 INFO - Driver provider org.openqa.selenium.edge.EdgeDriver registration is skipped:
registration capabilities Capabilities [{browserName=MicrosoftEdge, version=, platform=WINDOWS}] does not match the current platform MAC
17:40:18.316 INFO - Driver class not found: com.opera.core.systems.OperaDriver
17:40:18.316 INFO - Driver provider com.opera.core.systems.OperaDriver is not registered
17:40:18.318 INFO - Driver class not found: org.openqa.selenium.htmlunit.HtmlUnitDriver
17:40:18.318 INFO - Driver provider org.openqa.selenium.htmlunit.HtmlUnitDriver is not registered
17:40:18.356 INFO - Selenium Grid node is up and ready to register to the hub
17:40:18.376 INFO - Starting auto registration thread. Will try to register every 5000 ms.
17:40:18.376 INFO - Registering the node to the hub: http://localhost:4444/grid/register
17:40:18.395 INFO - The node is registered to the hub and ready to use

7. Now register a node to hub B as follows,

java -jar selenium-server-standalone-2.53.1.jar -role wd -hub http://localhost:4445/grid/register -port 5556

17:42:49.494 INFO - Launching a Selenium Grid node
17:42:49.817 INFO - Java: Oracle Corporation 25.51-b03
17:42:49.817 INFO - OS: Mac OS X 10.11.6 x86_64
17:42:49.820 INFO - v2.53.1, with Core v2.53.1. Built from revision a36b8b1
17:42:49.862 INFO - Driver provider org.openqa.selenium.ie.InternetExplorerDriver registration is skipped:
registration capabilities Capabilities [{ensureCleanSession=true, browserName=internet explorer, version=, platform=WINDOWS}] does not match the current platform MAC
17:42:49.863 INFO - Driver provider org.openqa.selenium.edge.EdgeDriver registration is skipped:
registration capabilities Capabilities [{browserName=MicrosoftEdge, version=, platform=WINDOWS}] does not match the current platform MAC
17:42:49.863 INFO - Driver class not found: com.opera.core.systems.OperaDriver
17:42:49.863 INFO - Driver provider com.opera.core.systems.OperaDriver is not registered
17:42:49.864 INFO - Driver class not found: org.openqa.selenium.htmlunit.HtmlUnitDriver
17:42:49.864 INFO - Driver provider org.openqa.selenium.htmlunit.HtmlUnitDriver is not registered
17:42:49.896 INFO - Selenium Grid node is up and ready to register to the hub
17:42:49.916 INFO - Starting auto registration thread. Will try to register every 5000 ms.
17:42:49.917 INFO - Registering the node to the hub: http://localhost:4445/grid/register
17:42:49.934 INFO - The node is registered to the hub and ready to use

8. Now you have setup 2 grid servers and registered a node each.

9. Let us now start configuring the nginx with a round-robin load balanced configuration.

add the following config to http node in the nginx configuration file "/usr/local/etc/nginx/nginx.conf"

    upstream gridapp {
      server localhost:4444;
      server localhost:4445;
    }

in the above, gridapp is loadbalanced name

Then update the server that listens on 80 port as follows.
server {
         listen 80;
         location / {
           proxy_pass http://gridapp;
         }
    }

This means, when users access http://localhost:80, nginx proxy passes to loadbalanced servers (localhost:4444, localhost:4445) named with gridapp.

To Test:
From your browser, hit http://localhost:80 and observe different nodes displayed.




Wednesday, December 7, 2016

Download Previous versions of Chrome Browsers

 Chrome Browser Versions:  Here are the links to download the previous versions of Chrome Browsers

https://commondatastorage.googleapis.com/chromium-browser-continuous/index.html
https://omahaproxy.appspot.com/

Saturday, November 19, 2016

Create Your First Angular application with npm, bower, angularjs

This post assumes you have already installed nodejs, npm on your machines. To start with let us use the terminal/sublime text editor to create our app which shows our name on the webpage.

Follow the steps to create your simple page.
1. Launch the terminal and create a directory named <ajs> anywhere in your drive. I chose "/Users/Yuvaraj" location to create the <ajs> directory.

    /Users/Yuvaraj> mkdir ajs

2. Navigate to the ajs directory.
    /Users/Yuvaraj> cd ajs

3. Now interactively create a package.json file in ajs directory by running the following command,
    /Users/Yuvaraj/ajs>npm init
     we should see the following questions on running this command.
 
     This utility will walk you through creating a package.json file.

It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg> --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (ajs) 
version: (1.0.0) 
description: 
entry point: (index.js) 
test command: 
git repository: 
keywords: 
author: 
license: (ISC) 
About to write to /Users/Yuvaraj/ajs/package.json:

{
  "name": "ajs",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Is this ok? (yes)

4. After generating the package.json file. Let us install http-server, bower libraries using the npm and save these dependencies into the package.json by running the following command.
   /Users/Yuvaraj/ajs>npm install --save http-server
   /Users/Yuvaraj/ajs>npm install --save bower
  Observe that http-server, bower are automatically saved into package.json file in the dependencies section and a folder node_modules is created with the libraries installed.

5. Now create an app directory where we put all the application code.
     /Users/Yuvaraj/ajs>mkdir app

6. Initialise bower by running the following command.  Make sure we see bower.json after running the command in app directory.
     /Users/Yuvaraj/ajs/app>bower init

7. After initialising the bower, install the angular dependency and save to the bower.json.
    /Users/Yuvaraj/ajs/app>bower install --save angular
   
    We observe that a new folder bower_components gets created in app folder with angular dependency added.   

8. Now edit the package.json file by adding a start script into the script section.

     "scripts": {
  "start": "http-server ./app -a localhost -p 2222",
        "test": "echo \"Error: no test specified\" && exit 1"
      },

With this step we are done with the setup. Once the setup is completed we can start creating the angular modules and bind them with the html.

9. Now create a app.js file in the app folder and put the following content in it.
 
    angular.module('yuvi', [])
     .controller('TestController', function TestController(){
         this.name = "Yuvaraj";
    });
 
10. Create a index.html in app folder and put the following html in it.
   
      <html ng-app='yuvi'>
       <head>
          <script type="text/javascript" src="/bower_components/angular/angular.js"></script>
<!-- this refers the angular.js file reference added from the bower dependencies installed -->
          <script type="text/javascript" src="app.js"></script>   <!-- this refers the app.js file that we have just created -->
        </head>

        <body ng-controller='TestController as testCtrl'>
            <h1>{{testCtrl.name}}</h1>
        </body>
     </html>

11. Now we are done with writing our simple app which displays our name. Now let us start our application by running the start script configured in our package.json.
 In the terminal run npm start command where the package.json file is available.

   /Users/Yuvaraj/ajs>npm start
   we must a message saying the server is started on a specific port with the url to the application.
    http://localhost:2222.

 Simply launch the url <http://localhost:2222> in the browser. We see "Yuvaraj" displayed as a header.
 

  

Friday, October 21, 2016

Launch Selenium Grid on Amazon EC2

Here are the steps to launch your selenium grid on ec2 instance.

1. To launch the selenium grid on ec2, you must have an aws console account.
2. Create a t2.micro instance through ec2 service in the console.
3. Click on Security groups link in the ec2 management console.
4. Click on the default/security group currently assigned to your instance.
5. Add the following protocol rules to the inbound and outbound traffic.
   Type: All TCP
   Protocol: TCP
   Port Range: 4444 / 0 - 65535 (default value)
   Source: 0.0.0.0/0
6. Using the keypair generated while creating the instance, connect to the box.
7. Create a folder <selenium>.
8. run the following command to download the selenium server jar file.
   wget http://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.1.jar
9. Launch the grid using the following command
     java -jar selenium-server-standalone-2.53.1.jar.1 -role hub
10. Now, open your browser on your local machine and launch the grid url in the browser
      http://xxxxxxxxx:4444/grid/console

Next post will describe the steps to register the nodes to the grid....

Friday, September 16, 2016

How to Install Groovy Plugin in the Jenkins and write the groovy code in the console



Install the groovy plugin:

In order to install the groovy plugin in your jenkins server, first we download the the required (groovy) plugin from the plugin repository https://updates.jenkins-ci.org/download/plugins/groovy/.

The plugin extensions are hpi. Once the plugin is downloaded from the plugin repository, go to Manage Jenkins page and Manage Plugins page. Now goto Advanced tab, then chose the plugin file you have downloaded and click upload. This will install the groovy plugin on your jenkins server.

Write groovy code in the groovy script console.

 Refer the https://www.cloudbees.com/jenkins/juc-2015/presentations/JUC-2015-USEast-Groovy-With-Jenkins-McCollum.pdf

groovy Script vs System Groovy Script

The plain "Groovy Script" is run in a forked JVM, on the slave where the build is run. It's the basically the same as running the "groovy" command and pass in the script.

The system groovy script, OTOH, runs inside the Jenkins master's JVM. Thus it will have access to all the internal objects of Jenkins, so you can use this to alter the state of Jenkins

Where to store the script?

● Can put it in source and pull it from source
● Or you can put the script directly into the command box that is provided


Friday, March 25, 2016

Configure Protractor


Follow these steps to configure the protractor.

1. install npm from https://nodejs.org/download/release/npm/
2. install protractor using "npm install -g protractor"
3. install webdriver using webdriver-manager update.
4. start the selenium server using webdriver manager
     webdriver-manager start
5. Now protractor will accept the configuration file (protractor.conf.js).

   exports.config ={
         seleniumAddress: 'http://localhost:4444/wd/hub'
         jasmineNodeOpts:{
                  showColors: true,
                  defaultTimeoutInterval: 3000  

           }
     }

6. protractor can start the selenium server if we provide the path to the selenium server standalone jar. This will slow down the overall time it takes to run the tests due to protractor having to start the server as opposed to an instance already running:
seleniumServer: './node_module/protractor/selenium/selenium-server-standalone-2.42.0.jar'

https://github.com/angular/protractor/blob/master/docs/api.md

Sunday, January 31, 2016

SOAP mocking SOAPUI Groovy XML Parsing


Mocking is one of most regular tasks that testers need to do as part of their Job. Handling the mock using groovy bit challenging. Following is an example to parse the request xml and evaluate the xpath on the request xml.

Example:
import java.text.*
import groovy.lang.*
import java.util.*
import com.eviware.soapui.support.XmlHolder

def sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSXXX")
def currentOrderId = new Date().getTime()
def holder = new XmlHolder(mockRequest.requestContent)
holder.namespaces['ns1'] = "http://xxxxxx/aaaaa/fffffffffffff/2.0/vvvvvvvv/4.0"
holder.namespaces['ns'] = "http://xxxxxx/aaaaa/ffffffffffffff/2.0/bbbbbbbb/4.0"
def nodeV2 =  holder.getDomNodes("//*")[3]
log.info nodeV2.getNodeName()
def hubId= nodeV2.attributes.getNamedItem("Hub").getNodeValue()

if(hubId.equals(400)){
context.setProperty("request1", hubId)
}


//def nodeV =  holder.getNodeValue("//ns1:OrderID")
//def nodeV =  holder.getDomNode("//ns1:OrderID")
//log.info nodeV.attributes.getNamedItem("id").getNodeValue()

Thursday, July 9, 2015

Sunday, June 14, 2015

References

 <!--https://cdivilly.wordpress.com/2009/07/20/maven-jruby-and-rails-migrations/-->
                            <!--http://www.weblogism.com/item/344/call-rake-from-maven-->
                            <!--http://www.ning.com/code/2011/09/jruby-sinatra-web-service-as-an-executable-jar/-->

 <!--http://chris.chowie.net/2011/08/28/Sinatra-with-JRuby-on-Heroku/-->

Wednesday, May 20, 2015

Sunday, November 30, 2014

Transforming a line item



Extending the World class is one way to extend the functionality of Cucumber. In this section we will
look at another. In a few of our step definitions we have specified that we wanted to use a specific
“line item”. In those steps we had to always take the String Cucumber passed and convert it to a
number using the to_i method. If we want Cucumber to automatically convert it for us we can use
a Transformation. Let’s look at that now.
                                        The first thing we need to do is create a new file in the support directory named transformations.rb. The file name doesn’t matter; I just like to collect all of my transformations in the same file. On a typically large project you might see up to five or six transformations. In this file we will create our first transformation:

  Transform /^line item (\d+)$/ do |line_string|
   line_string.to_i
  end


This structure should look familiar. It is very similar to step definitions except it begins with the
word Transform instead of Given, When, or Then. To break it down completely, what we are saying
with this code is find a capture group of a step definition (portion of a step definition surrounded by
parentheses) that contains “line item” plus a number and perform the following action on the value
of the number. In order for this Transformation to find the appropriate step definitions we will need
to move the opening parentheses to surround the entire match.

Then /^I should see "([^"]*)" as the name for (line item \d+)$/ do |name, lin\
e_item|

Now we no longer need to convert the line_item parameter to a number since the Transformation
takes care of this for us.

       This technique is very useful when you have values that need to be converted and you wish to move
the logic from multiple steps to one Transformation. For example, imagine you need to perform a
lot of date handling. It would be nice to say things like today, tomorrow, or yesterday and have a
Transformation object create a date object.

Source: Cucumber and Cheese book

Wednesday, September 3, 2014


How to automate testing of display of a site for different devices by means of Galen Framework

Refer: 

http://sysmagazine.com/posts/203506/

http://galenframework.com/docs/reference-galen-spec-language-guide/#Comments

Sunday, July 27, 2014

Generate Videos during Test Automation


I understand that videos play an important role in Test Automation. Some people think that videos
are not so much important in test automation, but my experience taught me that even videos give you lot of information that lets every one knows what happens during test execution, and other information that happens during test execution.

   Here, I would like to explain a very simple mechanism to have your scripts generate videos of your test scripts. We know that capturing videos are little tricky and need to depend on some third party tools. Here is a third party freeware/open source  using which we can have the videos generated for our scripts.
The example I describe would be in php as some people do not get an idea how to capture videos using php, as php is not flexible to use like other high level languages such as C#, Java, etc.,

FFMpeg is an command line tool using which we can generate videos. And I do not want to describe a lot here, to let readers do their task immediately.

download the ffmepg from the link: http://ffmpeg.zeranoe.com/builds/
once you download the build extract the file and put ffmpeg.exe in your system path you define in Environment variable. example: C:\DriverCapture is my path  i define in System environment varible "Path".
And then, write the command
command: ffmpeg -f dshow -i video=\"screen-capture-recorder\" -r 5 -threads 0 testMethod1.wmv
in your phpfunction. Now use shell_exec(command) to launch the video.

example: 
 public function StartVideoThreadForTest($testMethodName)
    {
        //Run the Command to Start running the video without CaptureVideo.exe
        $cmd="ffmpeg -f dshow -i video=\"screen-capture-recorder\" -r 5 -threads 0 ".$testMethodName.".wmv";
        shell_exec($cmd);
    }

Now call this method before your test runs. To stop the video capture use the following command to stop:
shell_exec('taskkill /F /IM ffmpeg.exe');

But, When you run your script you observe that your execution does not move from video launch command, to make it work.
Create a .NET console project and put the following code in the Main method.
namespace CaptureVideo
{
class Program
    {
        static void Main(string[] args)
        {
            Process ffmpeg = new Process();
            ffmpeg.StartInfo.FileName = "c:\\seleniumDrivers\\ffmpeg";
            ffmpeg.StartInfo.Arguments = " -f dshow -i video=\"screen-capture-recorder\" -r 5 -threads 0 -y "+args[0]+".wmv";
            ffmpeg.Start();
        }
    }
}
In the above program I put the ffmpeg in the c:\seleniumDrivers\ folder.
Compile the above program, you will see an exe (CompiledVideo.exe - project name) generated inthe debug folder. Noe put the compiled exe in the system path you used to place ffmpeg directory.
And now use the following php function to run the video recorder on background:

 public function StartVideoThreadForTest($testMethodName)
    {
        //Run the Command to Start running the video without CaptureVideo.exe
        //$cmd="ffmpeg -f dshow -i video=\"screen-capture-recorder\" -r 5 -threads 0 ".$testMethodName.".wmv";
        $path=__DIR__.'/../../../Output/';
        $cmd="CaptureVideo ".$path.$testMethodName."-".time();
        shell_exec($cmd);
    }

Tuesday, July 22, 2014


Download Auto IT installer and install it on your windows machine.

Write the following code to your Auto IT Script Editor.:

Code:
opt("MustDeclareVars", 1);0=no, 1=require pre-declare

Main()


Func Main()
  
    Local Const $dialogTitle = $CmdLine[2]
    Local Const $timeout = 5

    Local $windowFound = WinWait($dialogTitle, "", $timeout)
    
   
    $windowFound = WinWait($dialogTitle, "", $timeout)
    Local $windowHandle

    If $windowFound Then
       
        $windowHandle = WinGetHandle("[LAST]")
        WinActivate($windowHandle)
   
        ControlSetText($windowHandle, "", "[CLASS:Edit; INSTANCE:1]", $CmdLine[1])
        ControlClick($windowHandle, "", "[CLASS:Button; TEXT:&Open]")        
        
    Else
     
        MsgBox(0, "", "Could not find window.")
        Exit(1)
     EndIf
EndFunc    


And then Go to Tools and Compile which generates exe file that can be used in any language binding.

To upload multiple files in windows environment use the following example syntax :



C:\Users\User1Name\Documents>FileUploadScript.exe Open C:\Users\ddefilippo\Pict
ures\"{space}""'Image1"""{space}"""Image1- Copy"""