Does the View in database reduce the query performance?

Not really!! Here is why...

Few days back, I was speaking about one of our newly created Filter API with server side pagination to my friend. In the implementation we were querying the view to get the data. Since the pagination was at the server side, at a time the query will return only 10 records. His question was since views internally going to use the tables for querying, will it end up being less performative? Will it create the result set of all data and then pick the top 10 items?

My initial reaction was to agree with question and confess we have an issue. But I was sure that this can not be the case. If this is so then we could have had severe issues. I am not much of a DB guru myself so went and did some research with others to figure out the story behind the scenes. By the way we use MS SQL Server 2008 and I am not sure if this descriptions applies to other database or not. 

The question to ask was. We know that the views are used to simplify other queries or standardize access to data. but do we compromise on speed while doing so? 

I got a related answer from SO. When an SQL statement references a nonindexed view, the parser and query optimizer analyze the source of both the SQL statement and the view and then resolve them into a single execution plan. There is not one plan for the SQL statement and a separate plan for the view. So it is fine to use the views for my above use case.

Further, I got to know about Indexed views from this discussions, which can further improve the performance. Microsoft has a nice documentation on Improving Performance with SQL Server 2005 Indexed Views.

A MindMap for Java Developer Interviews

Over the years I have been a panelist in many of the interviews for Java Developers. I have previously written a post titled Top 7 tips for succeeding in a technical interview for software engineers which covers few of the general guidelines. In this post I will share a mind map containing general topics covered in a Java developer interview. I have prepared this as a general reference for myself to remember the pointers and to keep a common standard across the multiple interviews.

XMind gives a nice listing of the map. You can find the map here. Here is Image which you can download and use.

Finally here is a old fashioned tabbed content list which is easier to copy paste.

Java-Topics
OOPs
Encapsulation
Abstraction
Inheritance
Interface - Abstract Class
Casting
IS-A vs HAS-A Relationships
Aggregation vs Composition
Plymorphism
Method overloading vs Method Overloading
Compile time vs Runtime
Threads
Creating threads
Multitasking
Synchronization
Thread Transitions
Marker Interface
Serialization
Clonnable
Shallow copy vs Deep Copy
Collections
Map, List and Set
Equals - Hashcode
Legacy - Synchronized Classes
JVM
Stack vs Heap Memory
Garbage Collection
JRE, JVM, JDK
Class loaders
Exception
Checked Vs Unchecked Exceptions
Exception handling best practices
try, catch, finally, throw, throws
APIs
Files
String - StringBuffer - String Builder
Java IO
XML
SAX Based & DOM Based
JAXB - Java API for XML Binding
Access specifier 
Access modifier 
public
protected
deafult
private
final
static
synchronized
abstract
transient
volatile
 Inner/Nested Classes
JavaEE Basics
Packaging the Applications
WAR
EAR
Basics
MVC
Servlets
Listeners
Lifecycle
JSPs
APIs
JPA
JAX-WS
SOAP, WSDL Webservices basics
Contract first vs
JAX-RS
RESTful  and its advantages

JSF


This is a work in progress and I hope to refine it further. Let me know if you have any comments.

On the way to a blogger - My Journey with this blog so far

TalkNotes - How can Agile help you in clearing the technical debt?

This post is a write up to my talk titled "How can Agile help you in clearing the technical debt?" presented at the Scrum Bangalore Meetup 2013. It was a short talk for 30 Minutes, So thought of a write up to elaborate the ideas on the slides.

How can agile help you in clearing the technical debt from Manu Pk

Defining Technical Debt

As per Wikipedia, "Technical debt metaphor referring to the eventual consequences of poor or evolving software architecture and software development within a code base". Technical debt is most understood when it is compared with Cost of Change of software. More Debt means the cost of introducing a change in your system is more.

The term Technical Debt was coined by Ward Cunningham in the year 1992. He used the term to explain to his manager about a particular refactoring which he wanted to do. Since he was working in the financial industry it was easy to communicate in terms of debt. Ward explains more about the metaphor here. The term is immensely popular in communication of the importance of code refactoring to the business and stake holders. Like financial debt technical debt is some times unavoidable but we should take steps to minimize it.

As you can see in the above image when you have technical debt in your system you could either pay the interest or pay back the principle. When we do regular stories we do bit of "extra" work than what ideally needed due to the technical debt. This is an example of the interest payment. It is important that we take some time and effort to pay back the principle by refactoring and cleaning up of the code.

Technical Debt in Agile Context

The below principle from the list of agile principles was the best one I could find which refers to clearing technical debt "Continuous attention to technical excellence and good design enhances agility". In agile projects working software has a lot of emphasis but this in no way implies that the intrinsic value of software can be compromised. As we push for more and more features it is important that we take time to look back at the health of the application. So, make make sure that your agile projects do not become Fragile as it grows.

There are lots of ways to deal with technical debt. I want to talk about 3 simple steps that we have tried in our projects.

1. Tools - Your must have defense

One of the major improvement with the agile development practices is the widespread usage of continuous integration (CI). Your CI environment should be configured to measure the health of the application. It should include the report from static code analysis and test coverage applications. SonarQube is a god platform where you can create a dashboard and see results from the tools mentioned above. It also has a plugin to view the technical debt. Apart from the tools you should be using good programming practices and guidelines.

2. Negotiating with Product Owner

The second techniques that we have tried to minimize technical debt is to negotiate with the product owner and take him on board in the technical improvements that we wanted to do.  You have to Use the opportunity of Change requests to clean up the functionality. We had a module in our application with lots of bugs. There was a major change request which was planned on that module. After much discussions we identified that it would be much easier to rebuild the module than fixing the issues. Once we rebuild the change request was 75% easier to implement compared to the old code.

This means that we have to evaluate the possibility of refactoring with every opportunity and go for it when the time is right. It is the professional responsibility of a developer to communicate dept of technical debt in the application and we should take all steps possible to minimize it. 

We can also have technical Stories in Release planning. Contrary to a user story, a senior developer should describe the success criteria of the story. This is another case where you need to get your product owner on board.

3. Trying out new Stuff

In our projects most things can be done multiple ways and we have to choose one of them. There are also situations where we are not sure about the complexity of the task performed and story point estimation is difficult. We can use spike to deal with such situations. A spike is an experiment that allows developers to learn just enough about something unknown in a user story, e.g. a new technology, to be able to estimate that user story. A spike must be time-boxed. This defines the maximum time that will be spent learning and fixes the estimate for the spike.

A daemon thread is a low priority thread that is running on the operating systems. We have used the same strategy to do a major change in our application. It was upgrading our frameworks (Spring, Hibernate and Tapestry) to their latest versions. Since it is a time consuming activity, we do not want to stop everything else and work on it. So, we choose to keep doing it as a low priority tasks and it took us 10 sprints to complete. Since the team continuously delivered features during this period it was very easy to get the buy in from product owner.

Useful Links:-

1. Martin Fowler's Talk

2. Martin Fowler on Technical Debt

3. Agile Architecture Emerges or Does It

4. Using Spikes

5. Steve McConnell’s Talk

6. http://www.ontechnicaldebt.com

7. Ward Explains Debt Metaphor

Note:- There are many ways to define and minimize technical debt. I have discussed few ways that we have successfully tried. You can go through the above links to find more information and choose the best methods that suits you.

GOF Design Patterns Notes on Trello!

Recently I was reviewing Gang of Four Design Patterns and created a Trello board for the same. Well, Here is how it look like!

Hope it could be useful for others as well.

NoSQL Distilled : Book Review

I have finished reading the book titled "NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence" written by  Martin Fowler and Pramod J. Sadalage  recently and thought of a writing short review. First of all it short book and you can run through the pages fast. As the title says, the book is really an introduction of No SQL databases.

It was..

The number one reason for the use of NoSQL databases is not performance and should be our use case. This means if our data model does not fit well in to the relational model, rather than twisting the data we can choose alternate data stores. This is absolutely critical if you want NoSQL to be used in most of the mid sized enterprise applications.  Also companies will start to believe that they should consider NoSQL it even if they are not Google or Amazon. 

Normally the technology books written by its creators of it will only focus on the advantages of using it. It can be quite an issue if you do not do proper research before choosing your application architecture. Since both Martin and Pramod do not belong to this category, they does a fair evaluation of various choices available to us. The section which says where to use it and where not to use is particularly impressive on that front.

Even though the book is selling you a new idea of NoSQL databases, it does not plays down the significance and importance of existing RDBMS. The book also brings light to some of the fundamental trade offs in the world of data stores like  consistency, availability and performance.

and is also..

When I was reading the book I thought that Polygot persistence is about using multiple data stores in same application depending on the use cases. But authors also seems to suggest that applications should wrap around the data stores with services ( like web services or REST). If we need to do this ideally we could only have one data store accessed by one service. Which would conflict the first statement. I am not sure about the reverence this question but I would love to have read more about using the different data store in the same application and its pros and cons.

I mean to say, 

This is a very great book if you are  looking get in to wild jungle of NoSQL databases. The authors does a great job in giving the brief introduction and making sure the basics are covered.

Package your classes by Feature and not by Layers

Most of the enterprise Java applications share some similarities in their design. Mostly the packaging of these applications are driven by the framework used by them like Spring, EJBs or Hibernate etc. Alternatively you can group you packages by features. Like any other item regarding modeling this is not free from any issues. Lets discuss some trade-off and how get around them. In this post we will discuss the pros and cons of both approaches against common usage scenarios.

Package By Layer (PBL)

This is a the first thing that developers do when the create an enterprise application in to split it to number of layers like DAO, SERVICE, VIEW etc.. This gives nice separation of code when we use different frameworks at different layers. For example if I were to use Hibernate, Spring and JSF, then I will have all my Hibernate dependent code in the DAO layer and JSF related code in the VIEW Layer. This is sort of good in case I need to migrate to a new framework only at the view layer or DAO Layer.

Package By Feature (PBF)

Package-by-feature uses packages to reflect the feature set. It places all items related to a single feature (and only that feature) into a single directory/package. This results in packages with high cohesion and high modularity, and with minimal coupling between packages. Items that work closely together are placed next to each other. They aren't spread out all over the application.

This also increases coherence as a large percentage of a the dependencies of a class are located close to that class.

 

Comparing the approaches

Let me compare both the approaches in below dimensions.

1. Adding a new Feature.

In case of PBL code has to be added to VIEW, SERVICE and DAO Layers and it can be tedious. PBF solves this problem by grouping all the code related to same feature in to a single directory.

2. Changing a framework used.

As discussed already PBL makes it easier to change a framework as all the related code are kept at same place. Here we know exactly the scope of the change and its impact. In case of PBF we need to dig into all the feature set to see the framework related classes.

If you choose to migrate the framework module by module, then it could be argued that PBF is better than PBL.

3. Code Navigation.

As developers needs to work on the features most of the time using PBF is easier for code navigation. When you know exactly know what has to be done its not much advantage.

4. Keeping the Common Code.

Every application will have some components which will be reused across the features ie, the features are not always exclusive. In such case if we package all the features separately the interactions between them can be quite messy. As a general principle we need to reduce such interactions and increase the cohesion inside the package. We can get around the situation be adding such entities to a common package. This approach is used in many projects including Hibernate.

Conclusion

Most application will have 4 types of classes. ie,

1. Domain Objects

2. Business Services

3. Data Retrieval Logic

4. Data Representation Logic

If we use PBF it gives us a good structural representation but does not give any functional representation. We need our architecture to resemble the problem domain. So its better to use Package By Feature style. We could internally classify the classes by using naming conventions like MyFeatureDAO or MyFeatureService . This way we could communicate the intend of the classes.

I have found some discussions on this topic and hope that might also help you choose.

1. Java Practices Article

2. Lessons to Learn from the Hibernate Core Implementation

3. How Changing Java Package Names Transformed my System Architecture

4. Presentation By  Kostis Kapelonis

5. The Principles of OOD By Uncle Bob

6. Article says its Better to Use Package By Layer

7. Question on SO Is package by feature approach good?

A 3 Step Guide to Getting Started with NoSQL

I have been looking in to NoSQL databases for few months and would like to share my experience with it. This is a post might help you if you indent to start learning about the NoSQL Databases. I would try to link the resources which I found useful here.

Step 1: What is NoSQL?


NoSQL DEFINITION: Next Generation Databases mostly addressing some of the points: being non-relational, distributed, open-source and horizontally scalable. The original intention has been modern web-scale databases. The movement began early 2009 and is growing rapidly. Often more characteristics apply such as: schema-free, easy replication support, simple API, eventually consistent / BASE (not ACID), a huge amount of data and more. So the misleading term "nosql" (the community now translates it mostly with "not only sql").

As seen on NoSQL-Database.org.

Martin Flower's NoSQL page is a good starting point. His below talk on Goto Conference explains the need and structure of NoSQL data stores. Martin and Pramod has written a book titled  "NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence"  and is a good read. It summarizes his talks and other blog post into a book.  Martin has been an influential speaker on this topic and has written number of articles on this. I have read and seen many introductions but his work helped me to get things in to my head.



If you likes to view the slides, then the below presentation by Tobias Lindaaker on slideshare might inspire you. He gives similar ideas.

NOSQL for Dummies from Tobias Lindaaker

MongoDB has an online course MongoDB for Java Developers  which is really useful if you are interested in trying out things.

Step 2: How and for what are NoSQL used in Real world?

Once you have some idea, try to find the usage patterns. The above presentations will give lot of information on how these systems are used. You could go through the below links, which explains how specific business problems are solved using NoSQL. This is important because we could easily relate the case studies and get more insights into the capabilities of these systems.

1. MongoDB Customers page.
2. Powerd By Haddop 
3. Neo4J Customers Page
  
Step 3 : Find Usage Patterns that you could work on!

Once you have reached this point, you should try and implement the concepts. Look back at the application that you are working on and see if there is a need for an alternative data store. Do you store Product recommendations? Do you have issues with heterogeneous data? Can your application compromise ACID model for scalability? Do you store XML files or Images on you relational DB?  These are some of the questions that you could ask. This way you could determine if there is a serious need of a investigation for a alternative persistence mechanisms. This  is in no way means removing the RDBMS completely but moving to a polygot structure of data stores.

If there is no opportunity to try out these concepts in your work, you could create your own test projects and implement them. This way you would encounter problems and will learn from them.

5 Things a Java Developer consider start doing this year

This post is about 5 things that I am planning to do this year. I have created it for me to track my progress then thought it can be a good direction for anybody similar.

1. Create an application using a NoSQL Data store and connect it with Java

If you have yet not understood the NoSQL databases its the best time. Lots of choices on the NOSQL side, may be MongoDB or Hadoop can be a starting point. We can create applications using Spring Data or the native Java adapters to connect to the Data Store.

2. Get the first app on Java PaaS on Cloud and ask your 5 friends to use it.

You have many platforms available including Openshift backed by JBoss and Redhat , CouldFoundry backed by Spring source and VMware. Cloud is the future of application deployments and Software as service gaining more popularity. From a developer point of view nothing really changes apart from the configurations and deployment.

3. What really is Software Design?

Read the GOF Design Pattern catalog and Search your project for the usage of it. If you are not using them check if you have similar patters. If you have a Java enterprise application you can check for Java EE patterns. Take a existing use case and think of possible alternative implementations.

4. Learn a new Programming Language and create a sample project

I think here you have 2 broad choices, Ruby or a JVM functional language. There are a number of functional languages available. It will help you become a polygot programmer.

5. Contribute to the community

You should be doing it already if not it is the time to start. There are a number of ways including Community Forums, Stack overflow or write a blog

on how your leanings.

11 Online Learning websites that you should check out

Planning to start something new in this year? You can try online learning!!. Online education is gaining popularity over the last few years, as it should. I have tried few of them last year and will share my experience with them in this post. Since I am a software developer we are going to focus on the materials related to it.

So, here is a list of 15 sites that you should visit before you make you make your choice. I have not used much on all of them, but worth a spending some time. The italic text below the title are taken from the respective sites. The order of names are arbitrary and does not carry any significance.

1. MIT's OpenCourseWare

OCW makes the materials used in the teaching of MIT's subjects available on the Web.

Add caption

2. Coursera

We are a social entrepreneurship company that partners with the top universities in the world to offer courses online for anyone to take, for free. We envision a future where the top universities are educating not only thousands of students, but millions. Our technology enables the best professors to teach tens or hundreds of thousands of students.

3. Khan Academy

Learn almost anything for free

4. Google Developers

5. Code Academy

6. Code School

Code School teaches web technologies in the comfort of your browser with video lessons, coding challenges, and screencasts.

7. LearnStreet

Whether you are completely new to coding or a developer looking to learn a new language, LearnStreet courses make it engaging and fun to create with code. Our interactive courses are designed to help you learn by actually writing real code and getting immediate feedback.

8. wibit.net

WiBit.net is a video tutorial web site offering cutting edge programming and computer tutorials. We specialize in focused and linear content. WiBit is a great place to start learning how to program, or to pick up new skills even if you've been at it a while.

9. Interview street

Solve interesting challenges and also get connected to amazing tech companies if you are interested. Join in to be among the elite group of talent in the world.

10. PeepCode

Specialize in screencast tutorials for professional web developers and designers. Learn advanced Ruby, JavaScript, Ruby on Rails, Node, Git, Design, and connect with developers you admire in our popular Play by Play series.

11. P2PU.org

At P2PU, people work together to learn a particular topic by completing tasks, assessing individual and group work, and providing constructive feedback.

Related articles on the web:

1. http://mashable.com/2011/05/07/learn-to-code-online/

2. http://thenextweb.com/dd/2012/10/21/so-you-want-to-be-a-programmer-huh-heres-25-ways-to-learn-online/

3. http://www.makeuseof.com/tag/learn-code-10-free-fantastic-online-resources-hone-skills/

4. http://techcrunch.com/2012/08/21/5-ways-to-learn-code-from-the-comfort-of-your-own-browser/

5. http://thenextweb.com/dd/2012/01/21/7-ways-to-start-learning-how-to-code-right-now-for-free/