Archive

Posts Tagged ‘PHP Notes’

7 things every web developer should know about Linux administration

Web programming is not only about how you develop your applications, its also about how you deploy your application or website.Although control panels are being provided by almost every shared hosting company using which we can easily deploy our applications but as soon as application grows in terms of size or scalability we need to put that on virtual dedicated or dedicated or in a cloud server like amazon EC2 .

Linux system administration is specialized field in itself but if as a programmer you also know how to handle some common tasks you can get good turn around as well as can optimize your application for performance. Easy and efficient deployment process can also save your debugging and testing time. In this post I am going to list 7 Linux administration tasks which a programmer should know to be with ease while deploying or planning deployment of applications.

1. SSH connections
This is first and foremost requirement of Linux administration, you need to be able to login to remote system to do any kind of task on it. On windows putty or similar software can be used. On Linux or Mac OS machines you can directly use ssh command to do this. Telnet was once used for this purpose but nowadays it is outdated due to security issues. You can also make password less connection to your servers from your machines to automate many tasks. Read more about how to setup password less connections.

2. File Operations
You must be able to create,copy and move files and directories around your file system. Many people who are used to work on windows CMD prompt or Mac and Linux’s terminal will not face any problem in this. You just need to careful about using commands while doing operations which impacts files or directory life. Like using rm with rf option  with a small spelling mistake can erase your whole data without confirming.

3. Apache or your web server configuration
You must be aware of your web server’s functioning and its common configurations. Like on Apache you must be aware of virtual host settings and .htaccess file handling. You must be able to differentiate between ‘Dos’ and ‘Donts’. I can give you a point to ponder about. How using .htaccess file can slow down your overall file serving or execution via Apache and how you can overcome this? Try to found out information about it.You can also guide Linux Administrator about disabling modules which are not needed in your application to minimize memory foot print of the web server to boost overall performance.

4. Configurations
You must be able to see how your server is performing and how you can fine tune simple things like allocating more memory to your web or application server. If running PHP you must be able to handle common configuration of it as all configuration variables are not configurable through ini_set function or .htaccess file. Optimizing your database configuration can also be one of the useful thing to do.

5. Rsync
This is Swiss army knife for syncing your data between different servers. It only copy changed files and compression can also be used on the fly. You can start playing with it on your local machine and when you think that you can handle it, you will be surprised to see the impact it will make on your deployment process.

6. Using SVN, Git from command line.

You must be able to handle common operations related to SVN or Git through command line. This will help you to integrate your deployment process to your version system.

7. Using Cron Jobs

You must be able to set the schedules of your cron jobs. Using cronjobs you can handle routine work like clearing garbage or regenerating cache of specific portion of application.

 

I have tried to enlist some common tasks groups which you need to know to be better at linux administration as web developer. Some very common tasks like creating or editing files in ssh shell are also necessary in some cases.

How to identify Bigdata? A first hand explaination

Bigdata is the word which is being used very often but yet to be defined properly. How much data can be classified as big data? Does big data only means big in size or big in complexity or both? If hard disks can be clubbed together to store very very large amount of data then why whole world has become so obsessed with big data? Is it something different from traditional DBMS?

There are many questions which may come to the mind of a person who is starting his journey with big data. I can recall one of my friend Amrit who worked with a company which was also selling computers in year 1999. I asked him for quotes and he told me about a computer which was having 2GB of hard disk. He was very excited, in excitement he declare that you will not be able to fill that hard disk in next few years. In his words, “Its really Big hard disk which can hold big data” Today we can only laugh on it. Today even my car keys have a 64GB flash drive companion.

Cartoon: Big Data

Big is a relative term and its quite subjective. Although nowadays when data runs in several hundred GB we start calling it big data. My first encounter with big data was with apache logs on a server which was hosting more than few thousands websites. Due to restrictions on the number of opened file pointers on server, I tweaked apache configuration to store all logs in a single file with virtual host information as first column of the records.

We were supposed to process those logs for awstats log analyzer. Server was already under heavy load, so we transfer log files to another server and run our processing routines there. After processing data we were putting furnished awstats files back to originating server so users can see site stats without going to another server. For us this was big data.

I can identify following things due to which it was big for us :-

1. A single file was needed for multiple loggers as we were not able to write to multiple files at once due to restrictions on system.

2. As server was overloaded with high number of requests, we were not able to process our large file which require high memory as well as processor time on the same server.

So data can be classified as big data if your one machine is not able to create, hold or process it for the purpose you want to achieve.

If your machine is not sufficient to process or hold your data then first thing which come to your mind will be an upgrade of hardware. There is a catch in this option,  we have limitations on hardware upgrades and there will be a time when upgrades will not be possible.

So you need multiple systems to act like one. Whenever you feel that you are in that situation, you have big data at your disposal to handle.

Categories: PHP Notes Tags: ,

6 Algorithms you must know to be a good programmer

Algorithms are defined sets of instructions to solve a problem or achieve something. Almost every process in computer world can be classified as an algorithm, still it is most difficult part of computer science for many learners.

Algorithms are necessary to make your programs intelligent and efficient. They can make your program rock and it can stand out in front of competition. You need to be good at analysis, understanding and draw abstraction to get good hold on this subject.

In this post I am trying to compile a list of algorithms which every program should know. This list is not exhaustive and only indicative of some important sections of computer programming.

Sorting

Visualization of sorting algorithms experiments
You must be good at different kind of sorting algorithms like quick sort, bubble sort, merge sort, heap sort and many other useful sorting techniques. You must be able to differentiate between these algorithms based on their features, problems and potential usage scenarios. Try your hand with different combination of algorithms, who knows you can discover a new optimal algorithm while doing so.

Binary Search

binary search tree
This algorithm is necessary to be in sync with higher efficiency demand  which big data is presenting in front of programmers. Many databases are using this Binary search algorithm to provide quick query answers and it is also useful in many other scenarios.

Graph Search

Twitter social graph (social dances)

In this social network era, this algorithm is one of most needed and used algorithm. Many Algorithm like shortest path etc falls under this category. This category of algorithm is usually considered most difficult one to understand as it involves lots of dynamic variables. Knowledge of calculus in math really help in understanding this subject. Once you got the idea behind graphs, you can rock those optimal solution problems.

Stable Marriage

Do you know computer science can help you to find a good life partner or friend? This algorithm try to solve this problem in a given scenario where priorities of available people are dynamics and not linear in nature. This algorithm usually works to suggest good friends and life partner on may social networking or dating websites.

Map Reduce

MapReduce

Many algorithms has been developed to deploy MapReduce which is essential in developing distributed applications handling large data. This is essential part of Hadoop system using which we can run parallel processes to process a large amount of data.

Hilltop Algorithm

Many search engine marketers knows this algorithm which was initiated by Google to rank pages based on some kind of reference values. This algorithm (or similar algorithms) is necessary to dive into large amount of data which our world is generating everyday. Now we are facing a hill task to find useful data or pattern from so much of data noise, hill top can help in this.

 

There are many hundreds algorithms but you can start with these algorithms before you go to your journey of mastering this interesting subject.

 

One major thing which needs to be understood regarding algorithms are their patents. Many algorithms are patented and can not be used in our programs without a license.  So be sure that you are using an algorithm which you are entitled to use in your commercial programs for distributions. Many algorithms has been released under open source licenses.

How to optimize all databases and tabels on mysql server?

Optimization of database tables in mysql is a process which has to be carried out regularly to make your database in good health.

According to mysql manual,

OPTIMIZE TABLE should be used if you have deleted a large part of a table or if you have made many changes to a table with variable-length rows (tables that have VARCHAR, VARBINARY, BLOB, or TEXT columns). Deleted rows are maintained in a linked list and subsequent INSERT operations reuse old row positions. You can use OPTIMIZE TABLE to reclaim the unused space and to defragment the data file. After extensive changes to a table, this statement may also improve performance of statements that use the table, sometimes significantly.

In phpmyadmin you can see data similar to following screen if your database tables are not optimized. Entries in Overhead column indicates size of data space which can be reclaimed after optimization.

A major challenge comes in optimization when you have many databasesm thus many tables on my mysql server. It is hard to do optimization manually for each and every table. So we have created this script which can optimize each and every table of each database on your mysql server. You can run this script as your cron script also so you can do your optimization activities automatically. We have also executed repair statement which will repair any table error.


$con=mysql_connect(DATABASE_HOST,DATABASE_USER,DATABASE_PWD);
$db_list = mysql_list_dbs($con); $dbtoProcess=array();
while ($row = mysql_fetch_object($db_list))
{
$dbtoProcess[]=$row->Database;
}

foreach ($dbtoProcess as $dbname) {
$db=mysql_select_db($dbname);
$sql = "SHOW TABLES FROM `$dbname`";
$result = mysql_query($sql);

$AllTablesinDB=array();
while ($row = mysql_fetch_row($result)) {
$AllTablesinDB=array_merge($AllTablesinDB,$row);
}

foreach($AllTablesinDB as $value)
{
mysql_query("optimize table $value");
mysql_query("repair table $value");
//print $value ." has been optimized and repaired";

}
}

Execution of this script may take some time depending on the size of databases and tables.

How to integrate Magento and Drupal?

Magento is one of popular and robust e-commerce system which is based on the Zend Framework by Zend. It gives good tools and mechanism to operate your e-commerce websites and even multiple stores on different urls via single admin section.

Drupal is one of the best open source CMS having power of ease of use and lots of available modules which helps to build new functionality for users. It has powerful cck module for managing different meta values of a content entity. View helps to give representation of data in drag and drop kind of interface.

Now Magento and Drupal can be integrated with each other thanks to open source and free modules/extension which are available.

 

Drupal extension for Magento

Magento provides and extension for integrating Drupal into magento.

As per details given by the developer of Drupal extension :-

Drupal extension is a set of Core API extensions that offers a full access for Drupal modules  to Orders, Catalog and so on. This extension, will offer you to TOTALLY bypass Magento front office, using Drupal instead. When a new user is created a Magento user is automatically created also. Orders can be created from Drupal, cart is managed from Drupal. Catalog is syncrhonized, so each Magento Product is a Drupal node that you can push into Views, etc.

 

Magento module for Drupal

Drupal provides a Magento module which helps to import many magento entities in drupal as cck and you can also use views to customize your presentation layer.

 

As per details given by the development team at Drupal :-

  • Synchronization of Magento products to Drupal (currently supported type of products are: simple products, configurable products). All Magento attributes are dynamically synchronized into Drupal CCK fields
  • Custom product options
  • Up-sells, cross-sells, linked products
  • Synchronization of Magento categories into Drupal taxonomies
  • Synchronization of Drupal users to Magento customers, synchronization of addresses
  • Synchronization of currencies and currency conversion rates
  • Stock management
  • Full-featured shopping cart
  • Fully themable and customizable checkout process
  • Coupon codes
  • Payment modes API
  • Shipping methods API (support of all simple Magento shipping modes works out of the box)

 

 

Why People hate Magento?

Many people hate Magento and hate it upto a level where they really do not want to hear anything in favor of it. Most of them have some valid reasons (based on their personal experiences) and many of them hate it just to hate it.

On this page I am compiling a list of complains against Magento as an e-commerce system. I am not going to counter them or justify Magento in light of these complaints, I am compiling this list so you can take a balanced view before taking any decision in favor of or against Magento.

This list can also be used a ‘to do’ list by programmers and consultants to be claim expertise in Magento, because in my view any solution start from the “accepting” that their is some problem or we can say identification of the root cause of problem is necessary before giving any solution.

 I look in the database to find a centralized product table and I’m surprised no such think, oh, wait a denormalized flat table that may or may not be up-to-date

Magento is a huge dump of code. It reminds me a Windows. I really hate it.

The biggest problem about Magento is that it’s a black-box: it’s so huge and complicated, that’s it’s impossible to see what’s going on behind the scenes

slow slow slow slow slow slow slow slow ….

By default it’s just slow. If you’re a developer you’ll probably pull your hairs out if you still have any left after 2 days of coding and accomplishing nothing. Magento you suck!

EAV -> slow and complex. My first idea was to write a handy script reading all products from the old database and imports them into the one Magento uses. Didnt work out.

Rails, Symfony and all these great other web frameworks are convention based. Magento however is configuration based.

I will update this list regularly, even I am thinking about starting a new section where we can discuss possible solutions for some of these issues.

 

Just few days back I worked on website which was hit by a huge surge in search engine traffic and  we needed to put some special condition to make Magento’s session handling efficient. I will write about this in details in next article about Magento

How to optimize Magento speed?

I came across this good collection of some good suggestions by Guido Jansen about Magento speed optimization. Most of  suggestion are not exclusive Magento, but totality it is good list to optimize any Magento or PHP based application.  As of he has not finished his target of listing 101 suggestions, but a must read for anyone who wants to optimize Magento speed.

 

http://www.gxjansen.com/101-ways-to-speed-up-your-magento-e-commerce-website/

Deploy and Release your PHP Applications with Phing

Phing is a PHP project build tool based on Apache Ant. A build system helps you to perform a group of actions using a single command. If you’re wondering why PHP needs a build tool, consider a work flow where you write code and unit tests on your local machine, and if the tests pass you upload the code to staging/production server and make any changes to the production database. Without a build file, you’ll need to go through each step manually. If you are doing continuous integration, you’ll be doing the same steps over and over again. It’s too easy to accidentally omit something in the process and end up with serious problem in production.

 

Phing helps overcome such issues by automating tasks like running unit tests, applying database changes, deploying application code, etc.

 

In an article published on PHPMaster,Vito Tardia has given the details how you can use Phing to deploy our applications. Read complete article at PHPMaster

WordPress to Drupal Migration – Part 1

Recently I have done few projects where sites were growing rapidly in terms of functionalities and their current wordpress structure was not good enough to handle those changes. I am going to write about the summary of these personal experiences in this series of articles. I will summarize the steps and work I have done to do this migration.

Why Migration?

We should not do something because we can, should only do if need is there. So before starting it is necessary to understand if you really need the migration from WordPress to Drupal. WordPress is really a fantastic system and I have used the wordpress on website where we have done big customization and those sites were getting users visit in millions.

Following points can be considered and discussed with the team before finalizing migration :-
1. What is main purpose of migration?
2. What are the main problems which you are supposed to be solved with migration?
3. Is there any available way in the current system using which current system issues can be solved?
4. What is the time different between estimation of migration and solving problem within current system?
5. Do you see any future benefits in migration?
6. Do you have required skill set and resources to do the migration?
7. Are you migrating because you do not have resources to manage current system?

Gather answers for these questions and then analyze your situation.

In next part I will discuss what benefits Drupal can provide over a WordPress system?

What you should know in wordpress to be a wordpress programmer?

I got several messages that I should point out a list of topics which a developer should know in wordpress to categorize him/her as a wordpress programmer.

“WordPress Programmer” or “WordPress Developer” terms can be new to someone but due its popularity, term is getting attention and several companies are doing hiring based on wordpress experience only.

So in this post I am just enlisting some points which a wordpress programmer should know.

  • How to install wordpress? (Yes it really easy but you should know that :))
  • How to edit config file of wordpress if needed? You should also know what normally can be edited?
  • How to install a plugin in wordpress?
  • How to install a theme in wordpress?
  • How to modify a them in wordpress?
  • What are page templates?
  • How you can create/use a different template for archive or category page?
  • How you can use widgets?
  • How you can create a plugin?
  • How you can modify the content of a post or page?
  • How you can add something before or after a post/page content?
  • What are filters?
  • What are actions and how you can define callback functions on a particular action?
  • How you can create admin menu using plugins?
  • How you can edit the templates in online editor?
  • How you can manage widgets?
  • How you can manage users and thier different types?
  • How you can disable/enable comments on a wordpress site or on a specific page or post?
  • How you can use OPML files?
  • How you can import RSS feeds?
  • How you create custom meta tags in posts?
  • How you can add specific attributes to any post or page?
  • How you can mange user registration?
  • How you can create private blogs or posts?
  • At least 10-15 different plugins which can enhance/extend the basic wordpress functionality.
  • How you can use caching in wordpress?
  • How to upgrade wordpress?
  • How to take backup of wordpress posts using export?
  • How to take database backup of wordpress? (This can be classifieds as normall database backup process)

Above All

You should know what WordPress CAN not do by default and what we should not try to do with it