mars.learn.contrib.tensorflow.gen_tensorflow_dataset#

mars.learn.contrib.tensorflow.gen_tensorflow_dataset(tensors, output_shapes: Optional[Tuple[int, ...]] = None, output_types: Optional[Tuple[dtype, ...]] = None, fetch_kwargs=None)[source]#

convert mars data type to tf.data.Dataset. Note this is based tensorflow 2.0 For example ———– >>> # convert a tensor to tf.data.Dataset. >>> data = mt.tensor([[1, 2], [3, 4]]) >>> dataset = gen_tensorflow_dataset(data) >>> list(dataset.as_numpy_iterator()) [array([1, 2]), array([3, 4])] >>> dataset.element_spec TensorSpec(shape=(2,), dtype=tf.int64, name=None)

>>> # convert a tuple of tensors to tf.data.Dataset.
>>> data1 = mt.tensor([1, 2]); data2 = mt.tensor([3, 4]); data3 = mt.tensor([5, 6])
>>> dataset = gen_tensorflow_dataset((data1, data2, data3))
>>> list(dataset.as_numpy_iterator())
[(1, 3, 5), (2, 4, 6)]
Parameters

  • tensors (Mars data type or a tuple consisting of Mars data type) – the data that convert to tf.data.dataset

  • output_shapes – A (nested) structure of tf.TensorShape objects corresponding to each component of an element yielded from mars object.

  • output_types – A (nested) structure of tf.DType objects corresponding to each component of an element yielded from mars object.

  • fetch_kwargs – the parameters of mars object executes fetch() operation.

Return type

tf.data.Dataset