Upsert for delta data indexing..

[madhavtr]

Hi Team,

I am looking for a solution to index delta data for an existing index using Nest (.Net). While on searching I found the below code snippet for upsert..

client.Update<OldObjectType, UpdatedObjectType>(s => s
.Index("myindex")
.Id(id)
.Document(objins)
.Upsert(objins)
);

My questions are

1. in this update statement, do we need to have two different objecttype (old and updated) aka (T,K) ? what if the both the objects were same only its property is changed? for e.g, Car is the object .. color = "green" for old object.. and it has been updated as color = "forrest green".
2. I tried having both as same object. (T, T)..the upsert insert the same document twice rather than updating.
3. if i don't mention upsert in the above command.. there is no insert happening for the new document from the delta.

all sort of help are appreciated.

Regards
Madhavan.TR

[gmarz]

@madhavtr

Update is definitely one of the more complex APIs, so your confusion is totally understandable.

First off- from your example, it doesn't look like you're using the latest version of NEST (1.0). If possible, I would consider upgrading since we specifically changed the update API so that it would be less confusing. For instance, your example would become:

client.Update(s => s
    .Index("myindex")
    .Id(id)
    .Doc(objins)
    .DocAsUpsert()
);

Notice how this matches more closely to the actual elasticsearch update api.

I think the confusing part was that previously Document() did not pertain to doc in the elasticsearch API. Document() was only used to infer the document ID from your CLR type. This has been changed to IdFrom() in 1.0 GA. Have a look at #795 for some background on this.

To answer your first question: Yes, they should be different. T is the original type and K is a partial document containing the updated fields only (essentially what you would pass as the doc value in ES directly). In most use cases, it's probably best for K to just be object. For instance:

client.Update<MyPoco, object>(s => s
    .Index("myindex")
    .Id(id)
    .Doc(new { FieldToUpdate = "value" })
    .DocAsUpsert()
);

Hopefully this helps a bit.

[madhavtr]

Thanks for your response. I have downloaded the new Nest (1.0) dll.

I try to apply the update.. still it is getting inserted for the update..

Here is my code snippet.

1. OfficeLocations is the class that i have created.
2. I will be getting the updated/inserted delta data from stored procedure.
3. These data were converted into objects collections (List)
4. I am trying to update for the existing document and insert the new
   document.

if (client.IndexExists(i => i.Index("offlocations")).Exists)
{
foreach (OfficeLocations list in
officeLocations)
client.Update<OfficeLocations,
object>(s => s
.Index("offlocations")
.Id(list.jllid)
.Doc(new { FieldToUpdate = "value" })
.DocAsUpsert()
);
}

how do I update the document in this scenario?

Also, I could not able to index the documents using "CreateIndex" as per the documentation provided here http://nest.azurewebsites.net/ using 1.0.

Can I get any sample code for creating and updating documents (bulk) ?

Regards
Madhavan.TR

On Tue, Jul 29, 2014 at 6:01 AM, Greg Marzouka notifications@github.com
wrote:

@madhavtr https://github.com/madhavtr

Update is definitely one of the more complex APIs, so your confusion is
totally understandable.

First off- from your example, it doesn't look like you're using the latest
version of NEST (1.0). If possible, I would consider upgrading since we
specifically changed the update API so that it would be less confusing. For
instance, your example would become:

client.Update(s => s
.Index("myindex")
.Id(id)
.Doc(objins)
.DocAsUpsert()
);

Notice how this matches more closely to the actual elasticsearch update
api
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html#docs-update
.

I think the confusing part was that previously Document() did not pertain
to doc in the elasticsearch API. Document() was only used to infer the
document ID from your CLR type. This has been changed to IdFrom() in 1.0
GA. Have a look at #795
#795 for some
background on this.

To answer your first question: Yes, they should be different. T is the
original type and K is a partial document containing the updated fields
only (essentially what you would pass as the doc value in ES directly).
In most use cases, it's probably best for K to just be object. For
instance:

client.Update<MyPoco, object>(s => s
.Index("myindex")
.Id(id)
.Doc(new { FieldToUpdate = "value" })
.DocAsUpsert()
);

Hopefully this helps a bit.

—
Reply to this email directly or view it on GitHub
#834 (comment)
.

Regards
Madhavan.TR

[gmarz]

Hey @madhavtr, sorry for letting this issue slide a bit. That update code should work- check to make sure list.jllid contains a valid (existing) id, otherwise it would indeed treat the operation as an insert. Also, are you replacing new { FieldToUpdate = "value" } with the actual property and value you are trying to update?

The CreateIndex documentation is definitely stale, thanks for pointing that out, we'll correct it ASAP. Following the existing simple example from the docs, here's the correct usage:

var settings = new IndexSettings();
settings.NumberOfReplicas = 1;
settings.NumberOfShards = 5;
settings.Settings.Add("merge.policy.merge_factor","10");
settings.Settings.Add("search.slowlog.threshold.fetch.warn", "1s");

client.CreateIndex(c => c
    .Index("myindexname")
    .InitializeUsing(settings)
);

For examples on how to use the Bulk API, check out the recently updated section in the docs: http://nest.azurewebsites.net/nest/core/bulk.html.