What is robots.txt?

We all like search engines regularly crawl and index our website. But there are certain instances, when we want to hide some WebPages or information from the search engine bots. Actually we do not want to index those pages for security or any personal reasons. So we need a way to tell the spider not to crawl those pages or information that we want to hide from the rest of the world. Fortunately there is a way to do so is Robots Exclusion Protocol.

What is ROBOTS.txt

Definition:  Robots.txt is a simple text file that stops/controls the search engine bots/spiders from crawling selected web pages or files type.

robots.txt example

If there are particular files and folders that you do not want to be indexed by any search engine, you can use robots.txt. Though it is not mandatory but most of the search bots first check the robots.txt and follow the instruction on it. Having a robot file is a best practice for SEO also (We will discuss it latter). But it should not be considered as a replacement of any other security practices like password protection etc.


Placement of the robots file is very important. It should be in root directory(Example: http:/www.abcd.com/robots.txt) else search bots will not find it.


The structure of the robots file is pretty simple.  But you need to take care of misspelled use-agents, directory path, contradicting logical statements.

User-agent: *
# Allow all robots to visit the website


User-agent: *
Disallow: /
# Block all robots to visit the website


User-agent: msnbot
# Allow all robots from the website


User-agent: *
Disallow: /tmp/
Disallow: /logs # for directories and files called logs
# Block all robots from tmp and logs directories


User-agent: *
Disallow: /photos
Allow: /car/ mercedes.jpg
# This would tell Googlebot that it can visit ” mercedes.jpg” in the car folder, even though the “car” folder is otherwise excluded

Note: Directories and filenames are case-sensitive: “mycar”, “MYCAR”, and “Mycar” are all different to search engines.


User-agent: *
Crawl-delay: 10
# Crawl-delay parameter is used to set to the number of second’s crawlers to wait between successive requests to the same server.

Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
# Many crawlers also support a Sitemap directive, allowing multiple Sitemaps in the same robots.txt in the form

Robots.txt syntax

User-Agent:  -It the web crawler (e.g. “Googlebot,” “msn” etc.)
Disallow: / – Block the whole website for crawler
Disallow: /photos – Disallow the particular folder where as all the other directives are allowed.
Disallow: – Disallow none
Allow: – Allow all the pages and files in the web server.
Crawl-delay: 10 – sets to the number of seconds to wait between two successive crawl requests


The Robots Exclusion Protocol (RSP) is purely advisory. Though there are terms used disallow but it purely relies on the web crawlers. Robots do not guarantee the exclusion of the disallowed files/folders for all user agents. There are many malicious bots that do not obey the rule and intentionally visit the disallowed links.

Bottom line

No doubt robots.txt is a convenient way to block a specific page form search engines spiders. Effective usage of robots.txt is also essential for SEO too.  But you need to write it carefully as a faulty robots.txt file can affect you website.

What is .htaccess file?

I am sure as a blogger you all are aware of htaccess. The .htaccess is the abbreviation of Hypertext Access, and is a server configuration file that is mostly used by apache based web server. HTACCESS generally controls the directory it live in and the other directories underneath its parent directory.

In this article I’m going  to talk more about .htaccess  file which is a very important file both search engine optimization  and security of your website.

What is .htaccess file?


Supported by most web servers, htaccess file is a server configuration file that overrides the web server configuration for the directory that it is in, and all sub-directories within the directory.

HTACESS files are simple ASCII files with the extension .htaccess and no file name. Many people find it confusing for the strange naming structure but it is easy to create with any text editor.

The file remains hidden in your web server until you force your FTP client to show hidden files.

Sample .htaccess Code for wordpress

RewriteEngine On
RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
How to create .htaccess file?

I am not going to deep technical here.  Well if you are in windows you can create a text file  and name it as .htaccess . Remember there should not be any file name (filename.htacess). But it should be only .htaccess.

How to use the .htaccess file?

The .htaccess file is a very powerful tool when it comes to website security and search engine optimization. Things like Rewriting URLs, Customized error responses, Cache Control, Directory listing, server-side includes, blocking IPs can be performed by configuring the htacces files.

However any single mistake in writing the code can cause server errors and leave your site in not showing up. So you must take a great care while editing the codes.

How to edit .htaccess file?

As mentioned earlier, .htaccess files are text files and can be edited as a text file in general. But you must take a back up of the initial file before you upload the changed one.

So, you have it. Stay tuned for next part of the article where we will discuss more about the different htaccess codes to use for your specific requirement and SEO purpose.

Top Five SEO Trends for 2014

With ever rising internet users, Search Engine Optimization or SEO is now more significant for all most every business. From a small local restaurant to a international brand, to stay ahead in the market, and attain great success all you need to enhancing website visibility and appeal. For that reason, it is essential to analyze and learn more about the SEO trends for 2014.

how images improve your SEO

Creating High Quality Content

Google has made it very clear that it gives importance to content. No matter how big brand or company you are, if you aren’t creating high quality content, Google has no interest in you. It is also important that your content’s ultimate goal should be providing relevant information. Don’t stress on stuffing keywords. Users and Google are looking for answers. If you will give them what they are seeking, you will get a better ranking.

Better Assessments

It may not always be possible to assess which part of your advertisement strategy produced the most sales, whether it was emails, newsletters or Pinterest boards. So, new techniques are on the way to simplify and pinpoint the best tactics, as well as tactics which have been not received well. The Attribution Modeling Tool by Google tells you where first contact was made and when. This helps you to highlight all channels valuable to your conversion cycle.

Customer Focused Websites

If you want better conversions, you need to have a customer focused website. Make sure you are sending clear message to your customers. Make your website seamless, so the visitors can turn into customers. You need to have a proper sales funnel. Posting relevant content at the right time, which helps your customers, is another way of showing them that you care.

Social Media

Social Media is now an important part of SEO and you must start building a strong social presence for your website especially on Google+. You can Google+ authorship to your website to get maximum exposure.

Mobile Marketing

Teenagers and 20-somethings now use mobile technology far more than their predecessors, and they also use mobile devices more than PC’s or desktops. Business people also use mobile applications at a growing rate in order to stay connected to clients and co-workers on a fairly constant basis.

To make sure you connect with this market, make your site mobile-ready and have an m-commerce solution installed on your website so people can buy from you with their mobile devices.

How to Optimize a Landing Page for Your PPC Campaign?

Getting clicks on your ads are halfway to the success of your PPC campaign. The other half lies in the quality for the landing page that serves the ads. If the landing pages are not optimized well, you are definitely missing many of your potential conversions. So you need to put your best effort and time on landing page optimization.

Landing pages are the special web pages that come to view when we click on some advertisement. Landing page optimization is a part of conversion rate optimization in general where the goal is to improve the conversion or sale. Basically landing page optimization (LPO) aims to offer landing pages more appealing to the customers/visitors.

call to action

Let us find out the important factors of a conversing landing page. I have here listed the most important factors that make a good landing page:

  • Keyword Focused Headline
  • Short and Concise Content
  • Less Escape Route
  • Testimonials and Client Feedbacks
  • Social Signals
  • Ask for Minimum Information
  • Call-To-Action
  • Perform Testing

Keyword Focused Headline:

Probably the headline in a webpage is the first thing the visitor comes across when lands to a page. Once landed to the webpage visitors normally spend only 4-5 seconds in the landing page before he decides whether to stay or quite the page. So by providing clear headlines (keywords in the headlines) in your landing page make people to engage with your story. You can find more about how to write headlines here.

Short and Concise Content:

After clicking on the ads, visitors expect to land on a page that is relevant to their search query. So you need to direct them to a concise and relevant landing page. For instance a woman looking for “bikini tops”, if directed to the home page of the apparel store will never convert. Rather you need to send here to the bikini section of your store.

Make your landing pages specific. By specific, I mean the content of the landing pages must be relevant to your campaign keywords and short. Providing the visitors quick information regarding their search terms keep them engaged. Again the content hierarchy will help in better conversion as all the information in a landing page is not for every visitor.

Less Escape Route:

The landing pages are the entry point to the highly focused sales path. You need to remove any unnecessary site navigation or links in the landing page as they may act as distract. Without the excess baggage like navigation menus, links, advertisements, the visitors are left with only two options – either convert or leave the site.

Testimonials and Client Feedback:

This is an incredibly effective technique. Putting the customer testimonial and client logos can boost your conversion as they create a trust factor among the visitors. Though there is not any specific rule to follow on the placement of the testimonials, but I personally think the right sidebar is the best place to do so.

By putting a phone number in the landing page, it tells people you are legitimate enterprise and there are real people at the end of the line.

Social Signals:

Social signal plays a big factor in conversion. Strong social signal represents a solid reputation and trusted by most visitors.

Ask For Minimum Information:

This could be another deciding factor. Generally people do not want to share more information about them so you should not ask for information you don’t really need.

Call to Action:

Your traffic does not do any good until they convert to your customers. To the get most out of your landing page, the first thing to consider is your calls-to-action. Having an effective call to action is an essential part of any landing page. Again position of your call-to-action is an important factor.


Once you are done with a landing page design and content, the next part comes is testing. This is another area where you should put your sincere effort. The testing phase is most challenging and you need to test on hundreds of variables in the most common methods of testing that includes SPLIT TESTING, A/B testing, MULTIVARIATE testing. Effective tests will help you lift revenue.


Anything is possible, if executed correctly. If you are new to landing page optimizations, just keep your patient. You need any help, please feel free to shoot me an e-mail.