Lucene on clustered environment

I have a task to refresh lucene indexes periodically. However, when the application is deployed on a clustered environment, each node tries to start indexing process and others display the following error:

 Entity com.me.MyEntity Id 1783 Work Type org.hibernate.search.backend.UpdateLuceneWork  
 : org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/path_to_index/com.me.MyEntity/write.lock  
 at org.apache.lucene.store.Lock.obtain(Lock.java:84) [lucene-core-3.5.0.jar:3.5.0 1204425 - simon - 2011-11-21 11:20:12]  
 at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108) [lucene-core-3.5.0.jar:3.5.0 1204425 - simon - 2011-11-21 11:20:12]  
 at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:127) [hibernate-search-engine-4.1.0.Final.jar:4.1.0.Final]  
 at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:102) [hibernate-search-engine-4.1.0.Final.jar:4.1.0.Final]  
 at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:119) [hibernate-search-engine-4.1.0.Final.jar:4.1.0.Final]  
 at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99) [hibernate-search-engine-4.1.0.Final.jar:4.1.0.Final]  
 at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.1.0.Final.jar:4.1.0.Final]  
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_03]  
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_03]  
 at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_03]  
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_03]  
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_03]  
 at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_03]  

Those links may help you:
http://stackoverflow.com/questions/1228833/sharing-a-java-synchronized-block-across-a-cluster-or-using-a-global-lock 
http://stackoverflow.com/questions/17921898/using-lucene-in-a-clustered-environment-with-shared-nfs
http://forums.terracotta.org/forums/posts/list/6707.page
http://stackoverflow.com/questions/13486426/how-to-unlock-the-index-directory-in-lucene

But, i solved my case by adding a random timer to sleep :

 @Schedule(...)   
      public void myTask() {  
           try {  
                long sleepFor = (long)(Math.random() * 100000);  
                Thread.sleep(sleepFor); // hope nodes sleep different amount of time  
                log.infov("Sleep for this cluster : " + sleepFor);  
                Directory directory = FSDirectory.open(new File("path_to_index/com.me.MyEntity/"));  
             if (directory.fileExists("write.lock")) {  
               log.infov("ATTENTION!!! Already done by the other node!!");  
               return;  
             }  
             directory.close();  
 ...  

The random timer weakens the syncronization between nodes and the one who sleeps the least does the job! Then others know about this by checking the lock file..

Hope this helps..

lucene date issue

When indexing with Lucene (Hibernate Search), you may notice difference between dates in the database and indexed values. For example, in my case, Lucene dates were a day before the dates in the database. This is due to timezone differences. In the following annotation, org.hibernate.search.annotations.DateBridge assumes GMT time..

 @Column(name = "CREATED")  
 @Temporal(TemporalType.TIMESTAMP)  
 @DateBridge(resolution = Resolution.DAY)  
 private Date created;  

My solution was to write a custom field bridge named ESTDateBridge by copying the original DateBridge code and changing the default timezone from GMT to "America/New_York" (writing EST did not work..)
The working annotation is as follows:

 @Column(name = "CREATED")  
 @Temporal(TemporalType.TIMESTAMP)  
 @FieldBridge(impl = ESTDateBridge.class)  
 private Date created;  

Some references:
http://jamesjefferies.com/2009/04/22/hibernate-search-lucene-and-greenwich-mean-time/ 
https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_Data_Grid/6.4/html/Infinispan_Query_Guide/sect-Bridges.html 
https://forum.hibernate.org/viewtopic.php?f=9&t=990266&view=next 

a simple tutorial about joins

Here is an excellent tutorial about basics of joins: https://www.youtube.com/watch?v=mhs4vwTSY-U

In fact, videos of Oresoft LWC seems very good for self training :  https://www.youtube.com/user/TheOresoft/playlists

Oracle performance tuning by Guy Harrison

I watched a great lecture on Oracle performance tuning by Guy Harrison and wanted to share it with my notes. The lecture can be watched here:
https://www.youtube.com/watch?v=Ah1xjCl6Axg   (part 1)
https://www.youtube.com/watch?v=f-C3nqaqND0 (part 2)
.. and my notes taken while watching:

• application --(sql)--> database server --(request blocks)--> server memory --(request blocks)--> IO subsytem

          Problems in one layer may be caused or cured in the higher layer.

          Therefore, tune db top-down 

• app code
• sql
• contention
• memory
• disk IO

• Make Oracle work smarter, not harder.. do not load it too much (donkey example in the lecture..)
• Nulls can not be indexed, so null equality queries work slower. Use some predefined value instead of nulls.
• Use concatenated index while searching multiple columns, reduces the IO requests drastically. but inserts get slower at the same time.
• Try to cache returned results, so do not go to database unless you need to.
• Use bind variables (otherwise -> parse overhead). use PreparedStatement in java.
• Minimize lock duration, use optimiztic locking
• Reduce network traffic by using array interface (setFetchSize) and stored procedure for complex sql s.
• help the optimizer (table statistics etc.)
• fix bad sql if optimizer makes suboptimal choice.
• Bad sql -> with highest CPU or IO consumption.
• V$SQL -> sys.dm_exec_requests
• order by, group by, join two tables without index -> performs a sort
• hash joins are more efficient than sort-merge
• not having enough memory to sort is bad..

shrinking tables to gain space in database (Oracle Sql Developer)

When you delete rows, the rows may not be physically deleted and you may not get the used space back.  The case is explained in the post: http://laurenthinoul.com/how-to-fix-ora-01654-unable-to-extend-index-in-tablespace/

You can run the following sql to compute statistics :

analyze table   compute statistics;


You can view the statistics as in the picture:

try this sql to shrink:

alter table  enable row movement;
alter table  shrink space;
alter table  disable row movement;

analyze table  compute statistics;
commit;


however, this may give the following error due to the reasons here: http://agstamy.blogspot.it/2009/02/ora-10631-when-trying-to-shrink-table.html

SQL Error: ORA-10631: SHRINK clause should not be specified for this object
 "SHRINK clause should not be specified for this object"
*Cause:    It is incorrect to issue shrink on the object
*Action:   Verify the object name and type and reissue the command

in this case, you may try by removing the indexes, shrinking the table and recreating the indexes.

btw, truncating the table gives you the space back!