Query with _source filter fails in version 5.x

[sherry-ger]

What kind an issue is this?

• Bug report/enhancement request.

Issue description

ES Hadoop connector version: 2.x

Query like the following works:

{
	"_source" : ["ID","Name"],
	"query": {
		"match_all": {}
	}
}

ES Hadoop connector version: 5.x

The above query no longer works although it is still a valid ES query.

Steps to reproduce

Code:

// load data
    val queryStr = "{ \"match_all\": {} }"
    val fieldsToLoad = Array("ID", "Name")
//    val qStr =
//      s"""
//        |{
//        |  "_source": [${fieldsToLoad.map(field=> "\"" + field + "\"").mkString(",")}],
//        |  "query": $queryStr
//        |}
//      """.stripMargin
    val qStr =
      s"""
         |{
         |  "_source": [${fieldsToLoad.map(field=> "\"" + field + "\"").mkString(",")}]
         |}
      """.stripMargin
    val loadedRDD = sc.esJsonRDD(s"${prefixIndexName}_*/taxis", qStr, esProps)
    assert(loadedRDD != null)
    assert(loadedRDD.count() == 3)
    loadedRDD.foreach( pair => {
      val esID = pair._1
      assert(esID != null)
 
      val recordJson = pair._2
      assert(recordJson.contains("\"ID\""))
      assert(recordJson.contains("\"Name\""))
      assert(!recordJson.contains("\"StartDate\""))
    })

Strack trace:
None

Explanation:

The _source clause is used to limit the fields being returned to optimize the payload.

Version Info

OS: :
JVM :
Hadoop/Spark:
ES-Hadoop : 5.0.1
ES : 5.0.1

[jbaiera]

I'll update the breaking changes page with a blurb on this. When we introduced the sliced scroll feature there was a large amount of code that was contributed to help with parsing and building queries. This code checks the provided query string, and if the JSON contains a query field, it grabs that and disposes of the rest.

I don't think this is really classifiable as a bug per se, since the query support should really only be used to specify the query portion itself. Another thing here is that setting that field can interfere with how the framework populates data for schema based integrations like SparkSQL and Pig; The connector chooses values for that request parameter based on the names and types of the schema it is working with.

I think the correct course of action is to expose a new setting that would allow a user to set the contents of the source field for the request, but warn or throw in the event that those settings would collide with regular connector functionality (like in Pig or SparkSQL).