Skip to content

Implementation of Model, ModelFormat, TFLiteModelSource and subclasses #335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Aug 29, 2019

Conversation

ifielker
Copy link
Contributor

Implementation of Model, ModelFormat, TFLiteFormat, TFLiteModelSource, and TFLiteGCSModelSource

Copy link
Contributor

@hiranya911 hiranya911 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pointed out a few things that can be improved.

else:
self._data = {}
if display_name is not None:
_validate_display_name(display_name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can save a few lines if you get these validators to return the validated value.

self._data['displayName'] = _validate_display_name(display_name)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@hiranya911 hiranya911 assigned ifielker and unassigned hiranya911 Aug 27, 2019
Copy link
Contributor

@hiranya911 hiranya911 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty solid. Mostly just nits. However, consider this pattern when composing types:

class ModelFormat:
    def __init__(model_source, **kwargs):
       self._data = kwargs
       self.model_source = model_source

    def as_dict(self):
       # validate _data and model_source if necessary
       copy = dict(self._data)
       copy.update(self.model_source.as_dict())
       return copy

class TFLiteGCSModelSource:
        def __init__(self, gcs_tflite_uri):
             self.gcs_tflite_uri = gcs_tflite_uri

        def as_dict(self):
             # validate uri
             return {'gcsTfliteUri': self.gcs_tflite_uri}

Pros:

  • Knowledge of individual fields get encapsulated in the corresponding subclasses. So it's a better use of polymorphism.
  • No explicit setters needed.

Cons:

  • Validation is deferred until as_dict() is called. So calling a constructor with invalid params won't raise any errors. But trying to call an RPC will trigger the error (we use this pattern in FCM and it works well in practice).

@hiranya911 hiranya911 removed their assignment Aug 28, 2019
Copy link
Contributor

@hiranya911 hiranya911 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good. Just a couple of ideas on how to simplify the big constructors.

"""A Firebase ML Kit Model object.

Args:
display_name: String - The display name of your model - used to identify your model in code.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove "String -" part.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

model_format: A subclass of ModelFormat. (e.g. TFLiteFormat) Specifies the model details.
kwargs: A set of keywords returned by an API response.
"""
def __init__(self, display_name=None, tags=None, model_format=None, **kwargs):
Copy link
Contributor

@hiranya911 hiranya911 Aug 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This constructor seems to be doing a lot. Can we simplify as follows?

def __init__(self, display_name=None, tags=None, model_format=None):
    self._display_name = None
    self._model_format = None
    self._tags = None
    self._data = {}

    if display_name is not None:
        self.display_name = display_name
    if tags is not None:
        self.tags = tags
    if model_format is not None:
        self.model_format = model_format

@classmethod
def from_dict(cls, data):
    model_format = None
    tflite_format = data.pop('tfliteModel', None)
    if tflite_format:
        model_format = TFLiteFormat(**tflite_format)
    model = Model(data.pop('displayName', None), model_format)
    model._data = data
    return model

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@property
def create_time(self):
"""Returns the creation timestamp"""
create_time = self._data.get('createTime')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self._data.get('createTime', {}).get('seconds')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@property
def update_time(self):
"""Returns the last update timestamp"""
update_time = self._data.get('updateTime')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

model_source: A TFLiteModelSource sub class. Specifies the details of the model source.
kwargs: A set of keywords returned by an API response
"""
def __init__(self, model_source=None, **kwargs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify here too by adding a from_dict() method that handles json dict extraction

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@hiranya911 hiranya911 removed their assignment Aug 29, 2019
Copy link
Contributor

@hiranya911 hiranya911 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM 👍

@hiranya911 hiranya911 removed their assignment Aug 29, 2019
@ifielker ifielker merged commit 4618b1e into mlkit Aug 29, 2019
@ifielker ifielker deleted the mlkit-5 branch August 29, 2019 21:03
lahirumaramba pushed a commit that referenced this pull request Apr 20, 2020
* Introduced the exceptions module (#296)

* Added the exceptions module

* Cleaned up the error handling logic; Added tests

* Updated docs; Fixed some typos

* Migrating FCM Send APIs to the New Exceptions (#297)

* Migrated FCM send APIs to the new error handling regime

* Moved error parsing logic to _utils

* Refactored OP error handling code

* Fixing a broken test

* Added utils for handling googleapiclient errors

* Added tests for new error handling logic

* Updated public API docs

* Fixing test for python3

* Cleaning up the error code lookup code

* Cleaning up the error parsing APIs

* Cleaned up error parsing logic; Updated docs

* Migrated remaining messaging APIs to new error types (#298)

* Migrated FCM send APIs to the new error handling regime

* Moved error parsing logic to _utils

* Refactored OP error handling code

* Fixing a broken test

* Added utils for handling googleapiclient errors

* Added tests for new error handling logic

* Updated public API docs

* Fixing test for python3

* Cleaning up the error code lookup code

* Cleaning up the error parsing APIs

* Cleaned up error parsing logic; Updated docs

* Migrated the FCM IID APIs to the new error types

* Introducing TokenSignError to represent custom token creation errors (#302)

* Migrated FCM send APIs to the new error handling regime

* Moved error parsing logic to _utils

* Refactored OP error handling code

* Fixing a broken test

* Added utils for handling googleapiclient errors

* Added tests for new error handling logic

* Updated public API docs

* Fixing test for python3

* Cleaning up the error code lookup code

* Cleaning up the error parsing APIs

* Cleaned up error parsing logic; Updated docs

* Migrated the FCM IID APIs to the new error types

* Migrated custom token API to new error types

* Raising FirebaseError from create_session_cookie() API (#306)

* Migrated FCM send APIs to the new error handling regime

* Moved error parsing logic to _utils

* Refactored OP error handling code

* Fixing a broken test

* Added utils for handling googleapiclient errors

* Added tests for new error handling logic

* Updated public API docs

* Fixing test for python3

* Cleaning up the error code lookup code

* Cleaning up the error parsing APIs

* Cleaned up error parsing logic; Updated docs

* Migrated the FCM IID APIs to the new error types

* Migrated custom token API to new error types

* Migrated create cookie API to new error types

* Improved error message computation

* Refactored the shared error handling code

* Fixing lint errors

* Renamed variable for clarity

* Introducing UserNotFoundError type (#309)

* Added UserNotFoundError type

* Fixed some lint errors

* Some formatting updates

* Updated docs and tests

* New error handling support in create/update/delete user APIs (#311)

* New error handling support in create/update/delete user APIs

* Fixing some lint errors

* Error handling improvements in email action link APIs (#312)

* New error handling support in create/update/delete user APIs

* Fixing some lint errors

* Error handling update in email action link APIs

* Project management API migrated to new error types (#314)

* Error handling updated for remaining user_mgt APIs (#315)

* Error handling updated for remaining user_mgt APIs

* Removed unused constants

* Migrated token verification APIs to new exception types (#317)

* Migrated token verification APIs to new error types

* Removed old AuthError type

* Added new exception types for revoked tokens

* Migrated the db module to the new exception types (#318)

* Migrating db module to new exception types

* Error handling for transactions

* Updated integration tests

* Restoring the old txn abort behavior

* Updated error type in snippet

* Added comment

* Adding a few overlooked error types (#319)

* Adding some missing error types

* Updated documentation

* Removing the ability to delete user properties by passing None (#320)

* Adding beginning of _MLKitService (#323)

* Adding beginning of _MLKitService

* Added License and Docstring

* Firebase ML Kit Get Model API implementation (#326)

* added GetModel
* Added tests for get_model

* Firebase ML Kit Delete Model API implementation (#327)

* implement delete model

* Firebase ML Kit List Models API implementation (#331)

* implemented list models plus tests

* Implementation of Model, ModelFormat, TFLiteModelSource and subclasses (#335)

* Implementation of Model, ModelFormat, ModelSource and subclasses

* Firebase ML Kit Create Model API implementation (#337)

* create model plus long running operation handling
* Model.wait_for_unlocked

* Firebase ML Kit Update Model API implementation (#343)

* Firebase ML Kit Create Model API implementation

* Firebase ML Kit Publish and Unpublish Implementation (#345)

* Firebase ML Kit Publish and Unpublish Implementation

* Firebase ML Kit TFLiteGCSModelSource.from_tflite_model implementation and conversion helpers (#346)

* Firebase ML Kit TFLiteGCSModelSource.from_tflite_model implementation
* support for tensorflow lite conversion helpers (version 1.x)

* Quick pass at filling in missing docstrings (#367)

* Quick pass at filling in missing docstrings

* More punctuation

* Modify Operation Handling to not require a name for Done Operations (#371)

* Firebase ML Kit Modify Operation Handling to not require a name for Done Operations
* Adding support for TensorFlow 2.x

* rename from mlkit to ml (#373)

* Adding File naming capability to from_saved_model and from_keras_model. (#375)

adding File naming capability for ModelSource

* Firebase ML Modify Operation Handling Code to match rpc codes not html codes (#390)

* Firebase ML Modify Operation Handling Code to match actual codes
* apply database fix too

* Mlkit fix date handling2 (#391)

* Fix create/update date handling
* Skip unrelated failing tests (until sync)

* Firebase Ml Fix upload file naming (#392)

* Fix File Naming

* Integration tests for Firebase ML (#394)

* Integration tests for Firebase ML

* Fixing lint errors for Py3 (#401)

* Fixing lint errors for Py3

* Removed dependency on six

* Fixing a couple of merge errors

* Modifying operation handling to support backend changes (#423)

* modifying operation handling to support backend changes

* Firebase ML Changing service endpoint (#421)

* Mlkit add headers (#445)

* add Headers

* fixed test (#448)

* Adding tensorflow and keras so we don't skip tests (#449)

* Adding tensorflow and keras so we don't skip tests
* Add additional instructions for integration tests for ml

Co-authored-by: Hiranya Jayathilaka <[email protected]>
Co-authored-by: Kevin Cheung <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants