Showing posts with label EclipseLink. Show all posts
Showing posts with label EclipseLink. Show all posts

Saturday, June 25, 2011

JPQL and joins

Have you ever asked yourself if JPQL (Java Persistence Query Language) queries written by you REALLY do what you want, or you just prepare them, commit to the code repository, and forget about them with a little help from pizza and beer ;) ...

Let's take a look at very simple example, we have Person entity:
@Entity
@Table(name = "PERSONS")
public class Person implements Serializable {

    private String firstName;
    private Long id;
    private Person spouse;

    @Column(name = "FIRST_NAME")
    public String getFirstName() {
        return firstName;
    }

    @Id
    public Long getId() {
        return id;
    }

    @OneToOne(optional = true)
    @JoinColumn(name = "SPOUSE_ID")
    public Person getSpouse() {
        return spouse;
    }
...
}
and we want to fetch all persons matching some criteria.

Suppose that PERSONS table has following entries:

ID SPOUSE_ID FIRST_NAME
1 NULL John
2 NULL Mary
3 4 Adam
4 3 Eva

We will build and execute very simple JPQL query:
Query query = entityManager.createQuery(
    "select person from Person person where person.spouse.firstName = 'Eva' or 1 = 1");
Assert.assertEquals("Result list is too small,", 4, query.getResultList().size());
Our JPQL query should return all persons from the database, because of the "1 = 1" expression in where clause. Let's try how it works with different JPA Providers.

EclipseLink (2.2.0) converts our JPQL query into following SQL query:
SELECT t1.ID, t1.FIRST_NAME, t1.SPOUSE_ID 
FROM PERSONS t0, PERSONS t1 
WHERE (((t0.FIRST_NAME = 'Eva') OR (1 = 1)) AND (t0.ID = t1.SPOUSE_ID))
which leads to assertion error:
java.lang.AssertionError: Result list too small, expected:<4> but was:<2>small 
Hibernate (3.4.0 GA) uses following SQL:
select person0_.id as id1_, person0_.FIRST_NAME as FIRST2_1_, person0_.SPOUSE_ID as SPOUSE3_1_ 
from PERSONS person0_, PERSONS person1_ 
where person0_.SPOUSE_ID=person1_.id and (person1_.FIRST_NAME='Eva' or 1=1)
which leads to the same assertion failure.

Only OpenJPA (2.1.0) uses SQL expected by us:
SELECT t0.id, t0.FIRST_NAME, t1.id, t1.FIRST_NAME 
FROM PERSONS t0 
    LEFT OUTER JOIN PERSONS t1 ON t0.SPOUSE_ID = t1.id 
WHERE (t1.FIRST_NAME = 'Eva' OR 1 = 1)
How can we force EclipseLink and Hibernate to use left join? We have to modify JPQL query in following way:
select person 
from Person person 
    left join person.spouse spouse 
where spouse.firstName = 'Eva' or 1 = 1
As you see I've added explicit left join and changed first where clause expression to use this join.
Now JPQL query returns all 4 records regardless of used JPA Provider.

Sunday, May 29, 2011

EclipseLink - JPA Queries optimization - @JoinFetch, @BatchFetch and Query Hints

@JoinFetch is very helpful when you try to minimize the number of SQL queries standing behind JPA query. I wrote about it in one of my previous posts: JPA Demystified (episode 1) - @OneToMany and @ManyToOne mappings, but it brings also some threats (see JPA - insert instead of update). This post will describe one of them, related to limiting number of results returned by JPA query.

Let's use same entities as in JPA Demystified (episode 1) - @OneToMany and @ManyToOne mappings. We have Employer:
@Entity
@Table(name = "EMPLOYERS")
public class Employer implements Serializable {
    ...
    private List<Employee> employees = new ArrayList<Employee>();
    ...
    @OneToMany(mappedBy = "employer", cascade = CascadeType.ALL)
    @JoinFetch(JoinFetchType.OUTER)
    public List getEmployees() {
        return employees; 
    }
    ...
}
and Employee:
@Entity
@Table(name = "EMPLOYEES")
public class Employee implements Serializable {
    ...
    private Employer employer;
    ...
    @ManyToOne(optional = false)
    @JoinColumn(name = "EMPLOYER_ID")
    @JoinFetch(JoinFetchType.INNER)
    public Employer getEmployer() {
        return employer;
    }
    ...
}
As you see, nothing special, well maybe except the @JoinFetch annotation on employees property of Employer.

We will use the query fetching only few employers from the DB:
    ...
    @PersistenceContext
    private EntityManager entityManager;
    ...
    TypedQuery<Employer> query = entityManager.createQuery("select e from Employer e", Employer.class);
    query.setMaxResults(5);    
    List<Employer> employers = query.getResultList();
    Assert.assertEquals(5, employers.size());    
    employers.get(0).getEmployees().get(0);
    ...
Suppose that we have 8 employers in the database, each of them have 5 employees. When you run the above code you'll receive: java.lang.AssertionError: expected:<5> but was:<1>. Why?

The truth is hidden in the dark ;) - JPA query will produce following SQL (target DB is MySQL):
SELECT 
    t1.ID AS a1, t1.BUSINESS_NAME AS a2, 
    t0.ID AS a3, t0.FIRST_NAME AS a4, t0.EMPLOYER_ID AS a5 
FROM EMPLOYERS t1 
    LEFT OUTER JOIN EMPLOYEES t0 ON (t0.EMPLOYER_ID = t1.ID) 
LIMIT 0, 5
As you see, @JoinFetch annotation forced EclipseLink to fetch employers and employees in one SQL query, but limiting number of records is performed the same way as for fetching the employers only, and therefore the result set will be limited to: Employer 1 - Employee 1, Employer 1 - Employee 2, ... , Employer 1 - Employee 5. This result set will be in turn converted by EclipseLink to one employer with 5 employees attached, and the result of the query will be 1 Employer, instead of expected 5.

Good Lord! Is there any hope for the people looking for some queries optimization?! - Of course my dear Watson :) - we can always replace one EclipseLink specific annotation (@JoinFetch) with another (@BatchFetch) or the JPA Query hints ...

Let's focus on the @BatchFetch annotation first. We will modify the Employer entity in the following way:
...
    @OneToMany(mappedBy = "employer", cascade = CascadeType.ALL)
    @BatchFetch(BatchFetchType.JOIN)
    public List<Employee> getEmployees() {
        return employees;
    }
    ...
The @BatchFetch annotation tells EclipseLink to fetch all related objects in one query, so instead of one query (when using @JoinFetch) we will have two (one for fetching employers, and second one for fetching the related employees). As you probably suppose, our code will work correctly this time, at least on the Java level ;) - because when we check the generated SQL, we will see:
SELECT ID AS a1, BUSINESS_NAME AS a2 FROM EMPLOYERS LIMIT 0, 5

SELECT t1.ID, t1.FIRST_NAME, t1.EMPLOYER_ID, t0.ID, t0.BUSINESS_NAME 
FROM EMPLOYERS t0, EMPLOYERS t2, EMPLOYEES t1 
WHERE ((t1.EMPLOYER_ID = t2.ID) AND (t0.ID = t1.EMPLOYER_ID))
As you see it's not the optimal choice - we fetch only 5 employers and ALL employees, from which only those who are bound to one of our 5 employers are used. Waste of DB and network resources :(

When you try EXISTS type of @BatchFetch the queries looks like this:
SELECT ID AS a1, BUSINESS_NAME AS a2 FROM EMPLOYERS LIMIT 0, 5

SELECT t1.ID, t1.FIRST_NAME, t1.EMPLOYER_ID, t0.ID, t0.BUSINESS_NAME 
FROM EMPLOYERS t0, EMPLOYEES t1 
WHERE (
    EXISTS (
        SELECT t2.ID FROM EMPLOYERS t2 
        WHERE (t1.EMPLOYER_ID = t2.ID)
    ) AND (t0.ID = t1.EMPLOYER_ID)
)
Still not the optimal choice, same as the above we fetch 5 employers and ALL employees :(

Only the IN type produces the SQL, which can be useful for us:
SELECT ID AS a1, BUSINESS_NAME AS a2 FROM EMPLOYERS LIMIT 0, 5

SELECT t1.ID, t1.FIRST_NAME, t1.EMPLOYER_ID, t0.ID, t0.BUSINESS_NAME 
FROM EMPLOYERS t0, EMPLOYEES t1 
WHERE ((t1.EMPLOYER_ID IN (1, 2, 3, 4, 5)) AND (t0.ID = t1.EMPLOYER_ID))
As you see @BatchFetch annotation give us possibility to fetch exactly 5 employers, and limit the number of required SQL queries to 2. There is one potential problem with IN type usage, which is related to limit in number of expressions for SQL IN clause for some databases (ex. Oracle - max 1000 expressions).

If you prefer, for some reason, to control the employees fetching on per-query basis, you have to switch from the annotations usage to Query Hints, for ex.
query.setHint(QueryHints.BATCH, "e.employees");
query.setHint(QueryHints.BATCH_TYPE, BatchFetchType.IN);
Few lectures for the dessert:

Sunday, April 10, 2011

JPA and unmodifiable collections

Patrycja Węgrzynowicz (see Yon Labs / Yon Consulting) had very interesting talk on 33rd Degree Conference this year called "Patterns and Anti-Patterns in Hibernate". Inspired by this talk, I decided to verify JPA providers behavior regarding unmodifiable Collections.

I will use the same Employer - Employee - Benefit model as in my previous JPA posts (ex. JPA Demystified (episode 1) - @OneToMany and @ManyToOne mappings).

Let's take a look at the Employer entity:
@Entity
@Table(name = "EMPLOYERS")
public class Employer implements Serializable {
    ...
    private List<Employee> employees = new ArrayList<Employee>();
    ...

    public void addEmployee(Employee employee) {
        employee.setEmployer(this);
        employees.add(employee);
    }
    ...
    @OneToMany(mappedBy = "employer", cascade = CascadeType.PERSIST)
    public List<Employee> getEmployees() {
        return Collections.unmodifiableList(employees);
    }
    ...
    public void setEmployees(List<Employee> employees) {
        this.employees = employees;
    }
    ...
}
As you see addEmployee and getEmployees method usage should protect the employees list from the modifying outside the Employer.

Let's use following test:
    @Test
    @Transactional(readOnly = false)
    public void test03() throws Exception {
        Employer employer = employerDAO.get(1L);
        assertEquals(3, employer.getEmployees().size());
        employerDAO.merge(employer);
    }
Nothing special, we fetch the Employer from database, doesn't change anything, and try to merge the state into database. Simple, isn't it? Well, when you try to use Hibernate as JPA provider you'll encounter:
java.lang.UnsupportedOperationException
 at java.util.Collections$UnmodifiableCollection.clear(Collections.java:1037)
 at org.hibernate.type.CollectionType.replaceElements(CollectionType.java:501)
 at org.hibernate.type.CollectionType.replace(CollectionType.java:574)
 at org.hibernate.type.TypeFactory.replace(TypeFactory.java:505)
 at org.hibernate.event.def.DefaultMergeEventListener.copyValues(DefaultMergeEventListener.java:392)
 at org.hibernate.event.def.DefaultMergeEventListener.entityIsPersistent(DefaultMergeEventListener.java:200)
 at org.hibernate.event.def.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:173)
 at org.hibernate.event.def.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:81)
 at org.hibernate.impl.SessionImpl.fireMerge(SessionImpl.java:704)
 at org.hibernate.impl.SessionImpl.merge(SessionImpl.java:688)
 at org.hibernate.impl.SessionImpl.merge(SessionImpl.java:692)
 at org.hibernate.ejb.AbstractEntityManagerImpl.merge(AbstractEntityManagerImpl.java:235)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
 at $Proxy29.merge(Unknown Source)
 at [...].dao.jpa.DefaultEmployerDAO.merge(DefaultEmployerDAO.java:22)
 at [...].dao.EmployerDAOTest.test03(EmployerDAOTest.java:76) 
Good Lord, why it is trying to clear the collection!? Patrycja mentioned in her talk, that when you return the unmodifiable view of collection, Hibernate will treat the property as "dirty" (because both collections are different as objects) and try to persist the changes - which in this case means deleting all collection elements and persist same collection elements again!! ...

OK, maybe OpenJPA will behave better ...
java.lang.IllegalAccessError: class org.apache.openjpa.util.java$util$Collections$UnmodifiableRandomAccessList$0$proxy cannot access its superclass java.util.Collections$UnmodifiableRandomAccessList
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
 at serp.bytecode.BCClassLoader.findClass(BCClassLoader.java:50)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)
 at org.apache.openjpa.util.GeneratedClasses.loadBCClass(GeneratedClasses.java:67)
 at org.apache.openjpa.util.ProxyManagerImpl.getFactoryProxyCollection(ProxyManagerImpl.java:363)
 at org.apache.openjpa.util.ProxyManagerImpl.newCollectionProxy(ProxyManagerImpl.java:189)
 at org.apache.openjpa.kernel.StateManagerImpl.newFieldProxy(StateManagerImpl.java:1824)
 at org.apache.openjpa.kernel.StateManagerImpl.newProxy(StateManagerImpl.java:1790)
 at org.apache.openjpa.jdbc.meta.strats.StoreCollectionFieldStrategy.load(StoreCollectionFieldStrategy.java:543)
 at org.apache.openjpa.jdbc.meta.FieldMapping.load(FieldMapping.java:934)
 at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.load(JDBCStoreManager.java:691)
 at org.apache.openjpa.kernel.DelegatingStoreManager.load(DelegatingStoreManager.java:117)
 at org.apache.openjpa.kernel.ROPStoreManager.load(ROPStoreManager.java:78)
 at org.apache.openjpa.kernel.StateManagerImpl.loadFields(StateManagerImpl.java:3047)
 at org.apache.openjpa.kernel.StateManagerImpl.loadField(StateManagerImpl.java:3121)
 at org.apache.openjpa.kernel.StateManagerImpl.beforeAccessField(StateManagerImpl.java:1606)
 at org.apache.openjpa.kernel.StateManagerImpl.accessingField(StateManagerImpl.java:1591)
 at org.apache.openjpa.enhance.RedefinitionHelper.accessingField(RedefinitionHelper.java:59)
 at org.apache.openjpa.enhance.[...]$entities$domain$Employer$pcsubclass.getEmployees(Unknown Source)
 at [...].dao.EmployerDAOTest.test03(EmployerDAOTest.java:75)
Nope :(. OpenJPA fails much sooner - it is unable to fetch the Employees list!

Our last hope - EclipseLink ... Works :) it performs appropriate selects, to fetch the entity, then on merge verifies that there were no changes in it, and simply does nothing.

Of course all those 3 providers work perfectly when you simply return the collection, not the unmodifiable view.

Monday, March 21, 2011

Useless Flow Managed Persistence

Spring Webflow has some interesting feature called Flow Managed Persistence. In short: it allows you to change some persistent entity during the flow, and merge the state of entity into persistence context at the end of the flow.

Sounds interesting, and is worth trying for sure :) - but you have to be aware, that Spring Webflow guys have overlooked something very important. The JpaFlowExecutionListener, which is involved in the Flow Managed Persistence handling, binds an Entity Manager instance to the flow scope, when a flow execution starts.

So what? - you may say :) - hm, let's look at the Spring Webflow documentation: "any objects stored in flow scope need to be Serializable" - and now back to the Entity Manager interface - nope - it doesn't extend Serializable, so we have no guarantee that it will be.

Indeed when you use the EclipseLink as JPA provider you'll get beautiful "SnapshotCreationException: Could not serialize flow execution; make sure all objects stored in flow or flash scope are serializable".

I've reported this problem in Spring Webflow JIRA - see JpaFlowExecutionListener shouldn't assume that EntityManager is Serializable

You may also find this problem in Community Forum - EclipseLink, Toplink NotSerializableException - mentioned first on August 28th, 2008.

I'm waiting eagerly for the fix :)

Tuesday, December 7, 2010

JPA - insert instead of update

Few days ago I was digging through the code in some project to find the reason why EclipseLink (2.1.1) is performing database insert instead of update. Let me say, that it was the hardest thing to find in my whole JPA-using developer career - so for the records, and to let you find it faster than me ;)

There was an association between entities Alpha and Beta, made in following manner:
public class Alpha ... {
    ...
    private Beta beta;

    @ManyToOne(..., optional = false)
    @JoinColumn(name = "beta_id")
    @JoinFetch(JoinFetchType.INNER)
    public Beta getBeta() { 
        return beta;
    }
    ...
}
I wrote about the @JoinFetch annotation in one of my previous posts already (see: JPA Demystified (episode 1) - @OneToMany and @ManyToOne mappings) - it is used here, to fetch the associated Beta along with the Alpha, in one query.

In the project analyzed by me, instance of Alpha was created at one point, edited in few steps, with persisting into database between the steps. Everything was fine after the first step. Alpha was written into database (first insert), but after the second step, primary key violation occurred each time I tested it (same entity was again inserted into DB, while it should be updated at this point).

My first hint was missing equals and hashcode methods on Alpha and Beta. I've added it, but it doesn't helped at all. So I meet with the best friend of all Developers - Debugger :) - and started to dig into the project and EclipseLink source code. At this point let me give you first suggestion - when you trace the JPA problems use the Debugger and verify performed database operations in parallel, because only both those things can give you full view of the problem lurking beneath the Application's surface ;)

Watching the SQL queries, I've found that EclipseLink is performing query fetching the Alpha along with the Beta (as directed by @JoinFetch annotation), right before the invalid insert. Matching this operation to the Java code, I've found that EclipseLink is trying to check what has changed in the Alpha entity, to perform only the required updates to the database. The SQL query was using inner join ...

Do you suppose what was wrong? :) - There was no Beta entity associated to the Alpha, and therefore this query - "select ... from Alpha inner join Beta ..." was returning no records at all. At this point EclipseLink was assuming that Alpha entity was not written into database yet, and tried to do it again - Alpha has already generated identifier (while first insert into database), and ... beautiful exception :)

What you should learn from the above situation is that optional attribute of association definition is informational only! - It's provider choice to throw an exception if the associated value will be null, or do something completely else, as you see EclipseLink developers are ignoring it silently :(

Last but not least - use @JoinFetch(JoinFetchType.INNER) if and only if you are absolutely sure, that associated entity will always exist - ex. when the database scheme doesn't allow other possibility. Don't trust JPA provider completely ;)