com.aliyun.odps.tunnel

Class TableTunnel.UploadSession

java.lang.Object

com.aliyun.odps.tunnel.TableTunnel.UploadSession

Enclosing class:

TableTunnel

public class TableTunnel.UploadSession
extends Object

UploadSession 表示向ODPS表中上传数据的会话，一般通过TableTunnel来创建。
上传 Session 是 INSERT INTO 语义，即对同一张表或 partition 的多个/多次上传 Session 互不影响。
Session ID 是Session的唯一标识符，可通过 getId() 获取。

UploadSession 通过创建 RecordWriter 来完成数据的写入操作。
每个 RecordWriter 对应一个 HTTP Request，单个 UploadSession 可创建多个RecordWriter。

创建 RecordWriter 时需指定 block ID，block ID是 RecordWriter 的唯一标识符,取值范围 [0, 20000)，单个block上传的数据限制是 100G。
同一 UploadSession 中，使用同一 block ID 多次打开 RecordWriter 会导致覆盖行为，最后一个调用 close() 的 RecordWriter 所上传的数据会被保留。同一RecordWriter实例不能重复调用 close().
RecordWriter 对应的 HTTP Request超时为 120s，若 120s 内没有数据传输，service 端会主动关闭连接。特别提醒，HTTP协议本身有8K buffer。

最后调用 commit(Long[]) 来提交本次上传的所有数据块。
commit 操作可以重试，除非遇到以下异常：

• CloseUploadSessionStatusConflictException
• DataCollisionException
• InternalServerError
• LOCAL_ERROR_CODE

tips：

• 不建议对每条数据打开一个 ReadWriter，容易影响后续使用性能；
• 建议用户代码缓存至少64M的数据后，再使用一个 RecordWriter 一次性批量写入。

Method Summary

Modifier and Type	Method and Description
void	commit() 不进行校验的会话提交
void	commit(Long[] blocks) 提交本次上传的所有数据块
org.apache.arrow.vector.types.pojo.Schema	getArrowSchema()
Long	getAvailBlockId() 多个线程中的 TunnelBufferedWriter 将通过这个接口获得写入的 blockId 为了防止 blockId 重复分配，对于 curBlockId 的访问必须加锁。
Long[]	getBlockList() 获取当前会话已经上传成功的数据块列表
Configuration	getConfig()
String	getId() 获取会话ID
String	getQuotaName()
String	getResource()
TableSchema	getSchema() 获取表结构
TableTunnel.UploadStatus	getStatus() 获取会话状态
boolean	isShouldTransform()
Record	newRecord() 创建临时Record对象
RecordPack	newRecordPack()
RecordPack	newRecordPack(CompressOption option) 新建一个 ProtobufRecordPack，数据压缩方式 option
RecordPack	newRecordPack(int capacity, CompressOption option) 新建一个 ProtobufRecordPack，预设流 buffer 大小为 capacity, 数据压缩方式 option
ArrowRecordWriter	openArrowRecordWriter(long blockId)
ArrowRecordWriter	openArrowRecordWriter(long blockId, CompressOption option)
ArrowRecordWriter	openArrowRecordWriter(long blockId, CompressOption option, long blockVersion)
RecordWriter	openBufferedWriter() 打开一个无压缩 TunnelBufferedWriter 用来写入数据
RecordWriter	openBufferedWriter(boolean compress) 打开 TunnelBufferedWriter 用来写入数据
RecordWriter	openBufferedWriter(CompressOption compressOption) 打开 TunnelBufferedWriter 用来写入数据
RecordWriter	openBufferedWriter(CompressOption compressOption, long timeout) 打开 TunnelBufferedWriter 用来写入数据
RecordWriter	openBufferedWriter(CompressOption compressOption, long timeout, TableTunnel.BlockVersionProvider versionProvider) 打开 TunnelBufferedWriter 用来写入数据
RecordWriter	openRecordWriter(long blockId) 打开RecordWriter用来写入数据
RecordWriter	openRecordWriter(long blockId, boolean compress) 打开RecordWriter用来写入数据
RecordWriter	openRecordWriter(long blockId, CompressOption compress) 打开RecordWriter用来写入数据
RecordWriter	openRecordWriter(long blockId, CompressOption compress, long blockVersion)
void	writeBlock(long blockId, RecordPack pack) 打开http链接，写入pack数据，然后关闭链接，多次向同一个block写入时会覆盖之前数据
void	writeBlock(long blockId, RecordPack pack, long timeout) 打开http链接，写入pack数据，然后关闭链接，多次向同一个block写入时会覆盖之前数据
void	writeBlock(long blockId, RecordPack pack, long timeout, long blockVersion)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

isShouldTransform

public boolean isShouldTransform()

getAvailBlockId

public Long getAvailBlockId()

多个线程中的 TunnelBufferedWriter 将通过这个接口获得写入的 blockId 为了防止 blockId 重复分配，对于 curBlockId 的访问必须加锁。

Returns:

分配到的 blockId

Throws:

TunnelException

commit

public void commit()
            throws TunnelException,
                   IOException

不进行校验的会话提交

Throws:

TunnelException

IOException

writeBlock

public void writeBlock(long blockId,
                       RecordPack pack)
                throws IOException

打开http链接，写入pack数据，然后关闭链接，多次向同一个block写入时会覆盖之前数据

Parameters:

blockId - 块标识

pack - pack数据

Throws:

IOException

writeBlock

public void writeBlock(long blockId,
                       RecordPack pack,
                       long timeout)
                throws IOException

打开http链接，写入pack数据，然后关闭链接，多次向同一个block写入时会覆盖之前数据

Parameters:

blockId - 块标识

pack - pack数据

timeout - 超时时间 单位 ms 仅对 ProtobufRecordPack 有效 <=0 无超时

Throws:

IOException

writeBlock

public void writeBlock(long blockId,
                       RecordPack pack,
                       long timeout,
                       long blockVersion)
                throws IOException,
                       TunnelException

Throws:

IOException

TunnelException

openRecordWriter

public RecordWriter openRecordWriter(long blockId)
                              throws TunnelException,
                                     IOException

打开RecordWriter用来写入数据

BlockId是由用户选取的0~19999之间的数值，标识本次上传数据块

Parameters:

blockId - 块标识

Throws:

TunnelException

IOException

openRecordWriter

public RecordWriter openRecordWriter(long blockId,
                                     boolean compress)
                              throws TunnelException,
                                     IOException

打开RecordWriter用来写入数据

Parameters:

blockId - 块标识

compress - 数据传输是否进行压缩

Throws:

TunnelException

IOException

openRecordWriter

public RecordWriter openRecordWriter(long blockId,
                                     CompressOption compress)
                              throws TunnelException,
                                     IOException

打开RecordWriter用来写入数据

Parameters:

blockId - 块标识

compress - 数据传输是否进行压缩

Throws:

TunnelException

IOException

openRecordWriter

public RecordWriter openRecordWriter(long blockId,
                                     CompressOption compress,
                                     long blockVersion)
                              throws TunnelException,
                                     IOException

Throws:

TunnelException

IOException

openBufferedWriter

public RecordWriter openBufferedWriter()
                                throws TunnelException

打开一个无压缩 TunnelBufferedWriter 用来写入数据

Throws:

TunnelException

openBufferedWriter

public RecordWriter openBufferedWriter(boolean compress)
                                throws TunnelException

打开 TunnelBufferedWriter 用来写入数据

Parameters:

compress - 数据传输是否进行压缩

Throws:

TunnelException

openBufferedWriter

public RecordWriter openBufferedWriter(CompressOption compressOption)
                                throws TunnelException

打开 TunnelBufferedWriter 用来写入数据

Parameters:

compressOption - 数据传输压缩选项

Throws:

TunnelException

openBufferedWriter

public RecordWriter openBufferedWriter(CompressOption compressOption,
                                       long timeout)
                                throws TunnelException

打开 TunnelBufferedWriter 用来写入数据

Parameters:

compressOption - 数据传输压缩选项

timeout - 超时时间 单位 ms <=0 代表无超时. 推荐值: (BufferSizeInMB / UploadBandwidthInMB) * 1000 * 120%

Throws:

TunnelException

openBufferedWriter

public RecordWriter openBufferedWriter(CompressOption compressOption,
                                       long timeout,
                                       TableTunnel.BlockVersionProvider versionProvider)
                                throws TunnelException

打开 TunnelBufferedWriter 用来写入数据

Parameters:

compressOption - 数据传输压缩选项

timeout - 超时时间 单位 ms <=0 代表无超时. 推荐值: (BufferSizeInMB / UploadBandwidthInMB) * 1000 * 120%

versionProvider - BlockVersion 提供者，为内部产生的 blockId 分别指定 block version, null 代表不使用此功能

Throws:

TunnelException

getArrowSchema

public org.apache.arrow.vector.types.pojo.Schema getArrowSchema()

openArrowRecordWriter

public ArrowRecordWriter openArrowRecordWriter(long blockId)
                                        throws TunnelException,
                                               IOException

Throws:

TunnelException

IOException

openArrowRecordWriter

public ArrowRecordWriter openArrowRecordWriter(long blockId,
                                               CompressOption option)
                                        throws TunnelException,
                                               IOException

Throws:

TunnelException

IOException

openArrowRecordWriter

public ArrowRecordWriter openArrowRecordWriter(long blockId,
                                               CompressOption option,
                                               long blockVersion)
                                        throws TunnelException,
                                               IOException

Throws:

TunnelException

IOException

commit

public void commit(Long[] blocks)
            throws TunnelException,
                   IOException

提交本次上传的所有数据块

blcoks表示用户记录的已经成功上传的数据块列表，用来与服务器端做完整性校验

Parameters:

blocks - 用户预期已经上传成功的数据块列表

Throws:

TunnelException - 如果提供的Block列表与Server端存在的Block不一致抛出异常

IOException

getId

public String getId()

获取会话ID

getSchema

public TableSchema getSchema()

获取表结构

getQuotaName

public String getQuotaName()

getStatus

public TableTunnel.UploadStatus getStatus()
                                   throws TunnelException,
                                          IOException

获取会话状态

Throws:

TunnelException

IOException

getConfig

public Configuration getConfig()

newRecord

public Record newRecord()

创建临时Record对象

创建一个临时的Record对象, 用于后续上传数据

Returns:

newRecordPack

public RecordPack newRecordPack()
                         throws IOException

Throws:

IOException

newRecordPack

public RecordPack newRecordPack(CompressOption option)
                         throws IOException

新建一个 ProtobufRecordPack，数据压缩方式 option

Parameters:

option -

Throws:

IOException

newRecordPack

public RecordPack newRecordPack(int capacity,
                                CompressOption option)
                         throws IOException

新建一个 ProtobufRecordPack，预设流 buffer 大小为 capacity, 数据压缩方式 option

Parameters:

capacity -

option -

Throws:

IOException

getBlockList

public Long[] getBlockList()
                    throws TunnelException,
                           IOException

获取当前会话已经上传成功的数据块列表

Throws:

TunnelException

IOException

getResource

public String getResource()