helpful awk command examples when working with Apache access logs

There are some utils that analyze access logs, one of such util is webalizer. But I always believe that getting to know you access log files is more appropriate when you want to do some more in depth analysis on something specific. There are several commands available such as grep, cut, sed etc that you can use for different scenarios, depending on what kind of information you actually need, but in this post I will touch base on awk command and working with access logs. awk is a pattern directed scanning and processing language. Very powerful language indeed. awk manual can be found here Here is an Apache Access logs example [crayon-553f01d7cffbf046331416/] The format of any entry above is shown below, columns are whitespace delimited [crayon-553f01d7cffde151956928/] To extract above columns individually we can use the following commands [crayon-553f01d7cffec134301572/]   You will see that above commands taking a space as a delimiter, we can change this however as shown below [crayon-553f01d7cfff8773307029/] Let's work on some scenarios now, say if you want to get list of unique ip addresses from your logs, you can run this command to get that info [crayon-553f01d7d0009707651884/] or if you want to see which IP addresses has been accessing a specific resources then you can use either of these commands [crayon-553f01d7d0013013384894/] Check if the requests are coming from an automated scripts When checking for automated scripts, we will check for an empty user agent value, generally these scripts won't send through a user agent information Here is the command that you can use [crayon-553f01d7d001e078738046/] To check how many times a resource has been requested You can use the following command [crayon-553f01d7d0028876056233/]   Identify issues with your web resources Generally we will be working for 404 errors, we can get this kinda report from Google Analytics as well but Google won't list internal linking … [Read more...]

How to activate or enable color themes in bash scripts or other languages in Nano editor

Bored of looking at that black and white text for a while? You would want to enable or activate color themes in Nano editor which is really easy to do. There are many languages supported for color theming within nano, such as C/C++ PHP Ruby SH CSS Man Perl Python TCL HTML etc First you will want to open nanorc file with the command shown [crayon-553f01d7d1c5f480044883/] Then you would want to find include file that holds theming information such as one shown below [crayon-553f01d7d1c6e494912506/] Mostly what you have to do is to uncomment this line so above will become this [crayon-553f01d7d1c79039218883/] You can edit this file too, open one of the file in edit mode with nano You will get something like this [crayon-553f01d7d1c83208989507/]     You can change above mentioned colors to your taste for different things such as keywords, strings etc. icolor is case in-sensitve matching for the provided pattern. For more information on nanorc files, you should check out this link http://www.nano-editor.org/dist/v2.1/nano.html#Nanorc-Files I activated colors for bash scripts, Text within my bash scripts under nano looks like this now I hope that this article will be helpful to you. leave your comments if you would like to share any additional information. Cheers       … [Read more...]

Installing New Comodo Positive SSL Certificate on Apache and OpenSSL on Centos, Fedora and RedHat

Hello, I had to buy a certificate and install it on my new server I was issued with a new certificate by Comodo under Comodo Positive SSL. To start with I think it doesn't matter which certificate you choose, end of the day they all follow pretty much same standards and the only thing that make a certificate expensive is a company or issuing authority such as thwate (expensive). I personally always choose a cheapest certificate on market. Anyway with latest certificate I received 4 files in an archive as shown below You could just install a certificate by adding [crayon-553f01d7d25b0835202679/] and it will work on some browsers but you would need intermediate cert as well. Here is what always gets me. With latest PositiveSSL They give us 3 files which we need to combine and feed it to [crayon-553f01d7d25c6037671247/] So to do that you will run this command [crayon-553f01d7d25d3230866889/] Considering that all you files are under /home/crt/2014 folder our final VirtualHost confg will look like this   [crayon-553f01d7d25e0908713571/] Once you made above change, don't forget to restart your apache service generally located as /etc/init.d/httpd restart on Centos and Red Hat OS If you have any comment or would like to point out any error please leave your comments. … [Read more...]

Installing Apache thrift, Cassandra 2.x and YACassandra-PDO on CentOS 6.5+

Hello, Well after a long time this is my in length article, the reason I am writing is because of all the stuff I have been gone through building thrift and cassandra-pdo on centOS 6.5 Versions I will be writing about installing latest version of thrift that at time of writing and RnD was 0.9.1, CentOS is version 6.5 64 bit, php 5.3.3,  Apache/2.2.15, Cassandra is at version 2.06 which is most stable version for production at the time of writing.   Prerequisite Most of the commands will need elevated privileges. You should prefix sudo with every command, I am doing it on test VM so doing it as privileged user root Packages wget, nano are available if not then yum install them as shown below. Of course if you use vi or other util other than wget to download files then skip this [crayon-553f01d7d3063552667725/] Ok so lets get started. When I started building thrift, or cassandra pdo I was getting all sort of errors such as [crayon-553f01d7d3072351929805/]   Install Cassandra Installing cassandra is the simplest one, We will be installing it from rpm provided by datastax Open datastax.repo for editing with following command [crayon-553f01d7d307d355409384/] paste following content in editor [crayon-553f01d7d3087035150766/]  Install cassandra [crayon-553f01d7d3098773290530/] Above command will also install openJDK but cassandra guys recommend that we should use latest java from Oracle so lets do that now Get into opt directory You can actually choose to be any other directory of your liking [crayon-553f01d7d30a3189505496/] Get RPM path for latest jre  from http://www.oracle.com/technetwork/java/javase/downloads/index.html I am using jre 8 from http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html   [crayon-553f01d7d30b2232051805/] output of java --version would be  something like [crayon-553f01d7d30be788834039/] Lets now install Development … [Read more...]

Config file “/etc/amavisd.conf” does not exist

Hi guys, A quick tip If you get this error either trying to showkeys  or any other action using amavisd You can quickly fix that Generally amavisd.conf file reside under /etc/ but on many distros it can be found here /etc/amavisd/amavisd.conf  So all you have to to make your amavisd commands work is to create a symbolic link as shown below [crayon-553f01d7d4798476489279/] Now if you run your command again it should work. e.g. [crayon-553f01d7d47a7579860585/] should succeed now. I hope this help. If you have anything else to add please leave your comments below. Cheers,   … [Read more...]

Linux Find files newer than certain date and time

Yeah I know generally we pretty much always want to find older files but there are some use cases where finding newer files than a certain date time is a requirement. Ok being no where near to your Linux expertise, I will show you how I do it, you can show your approach to this problem by leaving your comment First step Create a random file with a timestamp as shown below [crayon-553f01d7d4e69532338797/] Second Step We will now find our files that are newer than the timestamp of our dummy file as shown below [crayon-553f01d7d4e78851847252/] If you want to show timestamps too with the listed file here is a modification to above command Showing timestamps [crayon-553f01d7d4e82185298105/] As said this is just one of a few approaches that you can use. With newer version of *ix you should also be able to use [crayon-553f01d7d4e8d213713581/] command line switch. To get more details on find command either use >man find or browse to http://unixhelp.ed.ac.uk/CGI/man-cgi?find to find more options that you can use. I hope this helps. Also a very happy new year 2014 Cheers, … [Read more...]

Bash Script to send Email in Linux

Hi Guys, If you have mail command available then using it in a shell script is not that difficult, If a guy like me can do it anyone can I guess because I am not a native Linux man :) Anyway lets check how to do it [crayon-553f01d7d5702129485211/] I saved above as jasmail.sh As you will notice that I am using the heredoc syntax in the above script. There are sometimes that you want to send custom email messages so accepting the message from command line param makes sense so here is a little modification to the above script so that you can make use of it if you think that it could be useful Alternative [crayon-553f01d7d5712353943295/] So as you can see that in the MSG container I've use ${1} bit which means that the first command param becomes my email message as shown below [crayon-553f01d7d571d850870626/] That makes sense I guess. This is just a basic bash script, the Linux Gurus reading this post will have more to share if they wish to leave their feedback. One last bit, do not forget to give execute permission to the current user else you will not be able to use above bash script. I hope this will help or at least give you an indicator that will serve towards your final result/requirement Cheers … [Read more...]

bash script to monitor apache httpd service and automatically restart/kill if it hangs or not responsive

Hi Guys, My Apache Web-server was playing up a little bit so until I found the cause of it my priority was to keep it up and running so I wrote a bash script to keep an eye on httpd crashes Lets check it out First up is to pick a script name and named it as httpdmon.sh and it is stored in my /home/jas/ directory CronJob Next I would just set a cron job so that it runs every minute and restart/start the service if required. [crayon-553f01d7d5e67980673567/]  Basic Script [crayon-553f01d7d5e75535952411/] Above script is good and as you can see that I am using curl to shoot a request to my local website and if the request times out i.e. after 5 seconds then an attempt to restart the service is made Now thats Ok but I would also like to be notified about it and also would like to force kill httpd process if for some reason it hangs A little bit more added to above script [crayon-553f01d7d5e81828883232/] As you can see that above will send me a message, you can replace mail command with sendmail etc its up to you. Running that solved few of my issues until I figured out what is causing httpd to crash. If I found something interesting I will be sharing it with you guys. Please note: I am not a Linux Advanced user but just knows enough to do some basic to intermediate things. If you have something to add to would like to improve above script please leave your comments as it would help others. Cheers, … [Read more...]

Linux How to get total directory size

Hi Guys, Just a quick one here, If you want to know size of a directory/folder you can use du command which is short form to disk usage You can cd to that directory and run this command Forget about the fact that I am logged in as root, I was just doing some weird things. Option 1 [crayon-553f01d7d660f153926887/] Option 2 [crayon-553f01d7d661d269931581/] or [crayon-553f01d7d6626533659356/] Now the above commands will print fill directory size but just in case you want to list sub directory size separately then this is how you do it [crayon-553f01d7d6631475292930/]   You can also specify directory path as shown below [crayon-553f01d7d663b631483915/] Above will work on /home/web folder You can also get quite interesting but if all depends what you are trying to do just type [crayon-553f01d7d6645031800667/] to see what else you can do here is a quick link to man page available through http://unixhelp.ed.ac.uk/CGI/man-cgi?du Above listed is just to give you an indicator Linux is so wide with its commands that there could be another 10s of combination to achieve above results but I am not not aware of those 10s so I rely on the above mentioned commands :) I hope this helps Just in case you would like to add something to this post or want to correct improper use of a command please leave your comments Cheers … [Read more...]

Basic Shell script to automatically start NodeJS if for some reason it stopped

Hi Guys, Well this one is a really basic example of a shell script that I used before in one of my production NodeJS website called http://icompressjs.com Here is the bash script that I wrote to monitor my local node install [crayon-553f01d7d6f3e295939061/]     as you can see that I am running Node on port 8080 and I am using curl to send a request to my node server. If response is 200 that is good and it will print "All Working well" else it will try to start NodeJS server You can also plugin another line to send email to you when a server is launched again. Something like this [crayon-553f01d7d6f4e556359885/] put above line just under echo "Launching" Now the final step is to cron this our bash script. I named it as node_monitor.sh You will have to give your local user rights to execute this script, after doing that you are able to cron it as shown below [crayon-553f01d7d6f5a174546276/] Above lines says that every 5th minute run node_monitor.sh If the response is ok i.e. 200 then nothing will be done else there will be an attempt to start NodeJS server again You can improve this script with better error handling but for what I was trying to do this was sort of enough I hope this helps Cheers         … [Read more...]