Skip to content

Conversation

hanbj
Copy link
Contributor

@hanbj hanbj commented May 28, 2018

    val conf = new SparkConf().setAppName("EsTest").setMaster("local[*]")
    conf.set("spark.sql.shuffle.partitions", "6")
    conf.set("spark.default.parallelism", "6")
    conf.set("es.cluster.name", "common")
    conf.set("es.nodes", "localhost")
    conf.set("es.port", "9200")
    val sc = new SparkContext(conf)
    val sqlContext = new SQLContext(sc)
    val index = "le_plan_d/dpa"
    val query = "{\"query\":{\"match_all\":{}}}"
    val dataFrame = sqlContext.esDF(index, query)
    dataFrame.show(2)

My index has more than 600 fields. I execute a simple match_all query.

The wrong information is as follows:
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: An HTTP line is larger than 4096 bytes.
{"query":{"match_all":{}}}

So,Transfer to post mode.

@jbaiera
Copy link
Member

jbaiera commented May 31, 2018

There seems to be some failing tests. Can you make sure that the tests complete successfully for you and comment when you have the PR updated?

@hanbj
Copy link
Contributor Author

hanbj commented Jun 1, 2018

This test class is not well written. Because HTTP URL length is restricted within es.
It is necessary to access es cluster validation.

Copy link
Member

@jbaiera jbaiera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hanbj thanks for taking a look. I agree that the test case is a bit wonky. I'll go ahead and merge this, and make a commit to make sure that test is doing what its supposed to be doing.

@jbaiera jbaiera merged commit a3c0b3d into elastic:master Jun 14, 2018
jbaiera pushed a commit that referenced this pull request Jun 14, 2018
Source filtering with a large number of fields can produce an HTTP URL that is larger
than 4096 bytes. Moving the source filtering information to the request body should
fix these problems.
jbaiera pushed a commit that referenced this pull request Jun 14, 2018
Source filtering with a large number of fields can produce an HTTP URL that is larger
than 4096 bytes. Moving the source filtering information to the request body should
fix these problems.
@wonderfuljamesjin
Copy link

Hi @hanbj and @jbaiera

I am having the same issue, can I know how to fix this issue. thanks a lot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants